On 26 January 2014 02:25, Sean Dague <s...@dague.net> wrote: > Something that's currently just far to manual to figure out is how testr > is functioning in the parallel case, which tests are running on which > process, and when does one of the pipelines run out of tests. The testr > assumption that you don't want output unless there is a failure is a > really bad assumption, because it's important to know that things are > progressing, especially in complicated systems.
IMO There are three distinct use cases - we need to be able to tell what was concurrent (the saved stream should do that post-hoc). Secondly we need to identify hangs and so on - human scale. Lastly we need automation around identifying broken things... > In tempest we actually have a wrapper that gives us a stream. But it's > got a couple of fundamental flaws. > > First - it is very clear that the expected behavior of start / end of > tests doesn't work: > > 2014-01-24 00:49:59.005 | > tempest.api.image.v1.test_images.CreateRegisterImagesTest.test_register_image_with_min_ram[gate] > 2014-01-24 00:49:59.005 | > tempest.api.image.v1.test_images.CreateRegisterImagesTest.test_register_image_with_min_ram[gate] > ... ok subunit2pyunit just routes everything through a regular python 'unittest.TextTestResult' which doesn't know about timing data or concurrency. So we need to replace that. > Question #1: how do we either make start message be when things actually > start? or turn it off entirely? > > > Second - it's not entirely obvious how unbalanced the pipelines are, and > that would be extremely useful to see in the output. A starting point > for a UI would be to add... > > " " x (5 * pipelinenumber) > > To the test results. Then you'd have a visual waterfall to the front of > the tests, and could clearly see pipeline differences. would (worker-N) be sufficient? Or do you want *'s/x's specifically? > Question #2: where would this be injectable in the stream of testtools / > subunit / testr / unittest? Long term I defintely want a richer UI in testr, but right now the most appropriate thing to do is to prototype the UI you need locally. -Rob -- Robert Collins <rbtcoll...@hp.com> Distinguished Technologist HP Converged Cloud _______________________________________________ OpenStack-dev mailing list OpenStack-dev@lists.openstack.org http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
