Hi, I was also just planning to work on it before Stephan contacted Thomas to ask about this test.
Thomas, you are right about the structure, the test should also go into the `run-nightly-tests.sh`. What I was planning to do is a simple job that consists of a Kinesis consumer, a mapper that fails once after n records, and a kinesis producer. I was hoping that creation, filling, and validation of the Kinesis topics can be done with the Java API, not by invoking commands in a bash script. In general I would try to minimise the amount of scripting and do as much in Java as possible. It would also be nice if the test was generalised, e.g. that abstract Producer/Consumer are created from a Supplier and also the validation is done over some abstraction that lets us iterate over the produced output. Ideally, this would be a test that we can reuse for all Consumer/Producer cases and we could also port the tests for Kafka to that. What do you think? Best, Stefan > On 8. Nov 2018, at 07:22, Tzu-Li (Gordon) Tai <tzuli...@apache.org> wrote: > > Hi Thomas, > > I think Stefan Richter is also working on the Kinesis end-to-end test, and > seems to be planning to implement it against a real Kinesis service instead > of Kinesalite. > Perhaps efforts should be synced here. > > Cheers, > Gordon > > > On Thu, Nov 8, 2018 at 1:38 PM Thomas Weise <t...@apache.org> wrote: > >> Hi, >> >> I'm planning to add an end-to-end test for the Kinesis consumer. We have >> done something similar at Lyft, using Kinesalite, which can be run as >> Docker container. >> >> I see that some tests already make use of Docker, so we can assume it to be >> present in the target environment(s)? >> >> I also found the following ticket: >> https://issues.apache.org/jira/browse/FLINK-9007 >> >> It suggest to also cover the producer, which may be a good way to create >> the input data as well. The stream itself can be created with the Kinesis >> Java SDK. >> >> Following the existing layout, there would be a new module >> flink-end-to-end-tests/flink-kinesis-test >> >> Are there any suggestions or comments regarding this? >> >> Thanks, >> Thomas >>
