Hi Ashish, Looks like Ewen already hit the main points, but a few additions:
1. ducktape repo is here: https://github.com/confluentinc/ducktape ducktape itself will be pip installable in the near future, and Kafka system tests will be able to depend on a particular version of ducktape. 2. The reporting is nothing fancy. We're definitely open to feedback, but it consists of: - top level summary of the test run (simple PASS/FAIL for each test) - top level info and debug logs - per-test info and debug logs - per-test "service" logs gathered from each service used in the test. For example, if your test pulls up a Kafka cluster with 5 brokers, the end result will have the Kafka logs from each of those 5 machines. Cheers, Geoff On Thu, May 21, 2015 at 3:15 PM, Ewen Cheslack-Postava <e...@confluent.io> wrote: > Ashish, > > 1. That was the plan. We put some effort into cleanly separating the > framework so it would be reusable across many projects. > 2. I think you're seeing a test in progress where the final report hasn't > been created yet. If you visit one of the older ones you'll see it has a > landing page with links: > http://testing.confluent.io/confluent_platform/2015-05-20--001/ Apparently > we need to adjust when we update the 'latest' symlink. The logs that are > collected for tests are configurable, and service implementations include > sane defaults (so, e.g., you will always get the normal log file for Kafka, > but only get the data files if the test asks for them). > 3. No code coverage support. Haven't looked into it, so I couldn't comment > on how hard it would be to add. > > -Ewen > > On Thu, May 21, 2015 at 2:38 PM, Ashish Singh <asi...@cloudera.com> wrote: > > > Geoffrey, > > > > This looks great! > > > > A few questions. > > 1. Will ducktape be maintained separately as a github repo? > > 2. How easy is viewing the test results and logs. The link in KIP, > > http://testing.confluent.io/confluent_platform/latest/, lists a bunch of > > files and dirs. Could you add to KIP how the result and logs for the > tests > > will be organized. > > 3. Does it support code coverage? If not, how easy/ difficult would it > be? > > > > On Thu, May 21, 2015 at 2:03 PM, Geoffrey Anderson <ge...@confluent.io> > > wrote: > > > > > Great, I'll work on putting together a more detailed map of this > > > replacement process. > > > > > > On Thu, May 21, 2015 at 11:13 AM, Gwen Shapira <gshap...@cloudera.com> > > > wrote: > > > > > > > Love this idea :) > > > > > > > > I took a look at Ducktape API and it looks like a good fit - clean > API, > > > > extensible, easy to use and powerful enough for our use-case. > > > > > > > > Something I'd like to see as part of the KIP is a map of what > > system-test > > > > currently tests, which ones we want to replace and a JIRA for > replacing > > > > (possibly one for each group of tests). > > > > Basically, I know we all want to use the new system for new test > cases > > > > (upgrades, etc), but I really want to make sure we don't get stuck > with > > > > both systems forever. > > > > > > > > Gwen > > > > > > > > On Thu, May 21, 2015 at 9:01 PM, Geoffrey Anderson < > ge...@confluent.io > > > > > > > wrote: > > > > > > > > > Hi, > > > > > > > > > > Just kicking off the discussion thread on KIP-25 > > > > > > > > > > > > > > > > > > > > > > > > > https://cwiki.apache.org/confluence/display/KAFKA/KIP+-+25+System+test+improvements > > > > > > > > > > Thanks, > > > > > Geoff > > > > > > > > > > > > > > > > > > > > -- > > > > Regards, > > Ashish > > > > > > -- > Thanks, > Ewen >
