Yeah, I'm all for getting these to run more frequently and on lighter weight infrastructure. (By the way, I also saw the use of docker; I'd really like to get a "native" docker cluster type into ducktape at some point so all you have to do is bake the image and then spawn containers on demand.)
A few things. First, it'd be nice to know if we can chain these with normal PR builds or something like that. Even starting the system tests when we don't know the unit tests will pass seems like it'd be wasteful. Second, agreed on getting things stable before turning this on across the board. Confluent runs these tests nightly on full VMs in AWS and historically, besides buggy logic in tests, underprovisioned resources tend to be the biggest source of flakiness in tests. Finally, should we be checking w/ infra and/or Travis folks before enabling something this expensive? Are the Storm integration tests of comparable cost? There are some in-flight patches for parallelizing test runs of ducktape tests (which also results in better utilization). But even with those changes, the full test run is still quite a few VM-hours per PR and we only expect it to increase. -Ewen On Thu, Nov 3, 2016 at 11:26 AM, Becket Qin <becket....@gmail.com> wrote: > Thanks for the explanation, Raghav. > > If the workload is not a concern then it is probably fine to run tests for > each PR update, although it may not be necessary :) > > On Thu, Nov 3, 2016 at 10:40 AM, Raghav Kumar Gautam <rag...@apache.org> > wrote: > > > Hi Becket, > > > > The tests would be run each time a PR is created/updated this will look > > similar to https://github.com/apache/storm/pulls. Ducktape tests take > > about > > 7-8 hours to run on my laptop. For travis-ci we can split them in groups > > and run them in parallel. This was done in the POC run which took 1.5 > hrs. > > It had 10 splits with 5 jobs running in parallel. > > https://travis-ci.org/raghavgautam/kafka/builds/171502069 > > For apache projects the limit is 30 jobs in parallel and across all > > projects, so I expect it to take less time but it also depends on the > > workload at the time. > > https://blogs.apache.org/infra/entry/apache_gains_additional_travis_ci > > > > Thanks, > > Raghav. > > > > On Thu, Nov 3, 2016 at 9:41 AM, Becket Qin <becket....@gmail.com> wrote: > > > > > Thanks Raghav, > > > > > > +1 for the idea in general. > > > > > > One thing I am wondering is when the tests would be run? Would it be > run > > > when we merge a PR or it would be run every time a PR is > created/updated? > > > I am not sure how long do the tests in other projects take. For Kafka > it > > > may take a few hours to run all the ducktape tests, will that be an > issue > > > if we run the tests for each updates of the PR? > > > > > > Thanks, > > > > > > Jiangjie (Becket) Qin > > > > > > On Thu, Nov 3, 2016 at 8:16 AM, Harsha Chintalapani <ka...@harsha.io> > > > wrote: > > > > > > > Thanks, Raghav . I am +1 for having this in Kafka. It will help > > identify > > > > any potential issues, especially with big patches. Given that we've > > some > > > > tests failing due to timing issues > > > > can we disable the failing tests for now so that we don't get any > false > > > > negatives? > > > > > > > > Thanks, > > > > Harsha > > > > > > > > On Tue, Nov 1, 2016 at 11:47 AM Raghav Kumar Gautam < > rag...@apache.org > > > > > > > wrote: > > > > > > > > > Hi, > > > > > > > > > > I want to start a discussion about running ducktape tests for each > > pull > > > > > request. I have been working on KAFKA-4345 > > > > > <https://issues.apache.org/jira/browse/KAFKA-4345> to enable this > > > using > > > > > docker on travis-ci. > > > > > Pull request: https://github.com/apache/kafka/pull/2064 > > > > > Working POC: https://travis-ci.org/raghavgautam/kafka/builds/ > > 171502069 > > > > > > > > > > In the POC I am able to run 124/149 tests out of which 88 pass. The > > > > failure > > > > > are mostly timing issues. We can run the same scripts on the laptop > > > with > > > > > which I am able to run 138/149 tests successfully. > > > > > > > > > > For this to work we need to enable travis-ci for Kafka. I can open > a > > > > infra > > > > > bug to request travis-ci for this. Travis-ci is already running > tests > > > for > > > > > many apache projects like Storm, Hive, Flume, Thrift etc. see: > > > > > https://travis-ci.org/apache/. > > > > > > > > > > Does this sound interesting ? Please comment. > > > > > > > > > > Thanks, > > > > > Raghav. > > > > > > > > > > > > > > > -- Thanks, Ewen
