+1 On Fri, 9 Mar 2018 at 5:47 PM Aljoscha Krettek <aljos...@apache.org> wrote:
> Hi All, > > Stephan pointed this out the other day to me, so here goes: as some of > you might now, there are end-to-end tests in flink-end-to-end tests that > run a proper Flink cluster (on the local machine) and execute some tests. > This catches bugs that you only catch when using Flink as a user because > they exercise the whole system. We should add tests there that verify > integration with other systems. For example, there's a bunch of > Docker-compose configurations for starting complete Hadoop clusters [1] or > Mesos [2] and there is other files for starting ZooKeeper, Kafka, ... We > can use this to spin up a testing cluster and run Flink on YARN and Mesos > and have a reproducible environment. > > As a next step, we could perform the sort of tests we do for a release, as > described here: [3]. For example, the test where we run a job and mess with > processes and see that Flink correctly recovers and that the HA setup works > as intended. > > What do you think? > > By the way, I'm also mostly writing to see if anyone has some experience > with Docker/Docker compose and would be interested in getting started on > this. I would do it myself because having more automated tests would help > be sleep better at night but I'm currently too busy with other things. 😉 > > [1] > https://github.com/big-data-europe/docker-hadoop/blob/master/docker-compose.yml > [2] https://github.com/bobrik/mesos-compose > [3] -- Liu, Renjie Software Engineer, MVAD
