Hi Thomas, the community is really interested in adding an end-to-end test for the Kinesis connector (producer as well as consumer). Thus, it would be really helpful if you could contribute your work you've already done.
Using Kinesalite sounds good to me and you're right and that we assume that Docker is available in our testing environment. The testing job would go into a separate module as you've suggested (flink-end-to-end-tests/flink-kinesis-test) and the entrypoint to the test would go into flink-end-to-end-tests/test-scripts/ plus flink-end-to-end-tests/run-nightly-tests.sh. I think Stefan stopped his work on the end-to-end test but he had some ideas about reusing testing infrastructure for the Kafka and Kinesis tests (e.g. having a test base for similar connectors). This is something we can also address after the release if it would entail too much work. Cheers, Till On Thu, Nov 8, 2018 at 7:22 AM Tzu-Li (Gordon) Tai <tzuli...@apache.org> wrote: > Hi Thomas, > > I think Stefan Richter is also working on the Kinesis end-to-end test, and > seems to be planning to implement it against a real Kinesis service instead > of Kinesalite. > Perhaps efforts should be synced here. > > Cheers, > Gordon > > > On Thu, Nov 8, 2018 at 1:38 PM Thomas Weise <t...@apache.org> wrote: > > > Hi, > > > > I'm planning to add an end-to-end test for the Kinesis consumer. We have > > done something similar at Lyft, using Kinesalite, which can be run as > > Docker container. > > > > I see that some tests already make use of Docker, so we can assume it to > be > > present in the target environment(s)? > > > > I also found the following ticket: > > https://issues.apache.org/jira/browse/FLINK-9007 > > > > It suggest to also cover the producer, which may be a good way to create > > the input data as well. The stream itself can be created with the Kinesis > > Java SDK. > > > > Following the existing layout, there would be a new module > > flink-end-to-end-tests/flink-kinesis-test > > > > Are there any suggestions or comments regarding this? > > > > Thanks, > > Thomas > > >
