Hello everyone, Yes, I am working on this now. We have some success already, but there is a lot of work to do. Of course, some things don't work ideally. For example, in [2] from the previous letter we have not 24 skipped tests, actually much more. So we have a bug somewhere :)
Regards, Yaroslav Lobankov. On Thu, Nov 26, 2015 at 3:59 PM, Andrey Kurilin <akuri...@mirantis.com> wrote: > Hi! > Boris P. and I tried to push a spec[1] for automation tempest config > generator, but we did not succeed to merge it. Imo, qa-team doesn't want to > have such tool:( > > >However, there is a big concern: > >If the script contain a bug and creates the configuration which makes > >most tests skipped, we cannot do enough tests on the gate. > >Tempest contains 1432 tests and difficult to detect which tests are > >skipped as unexpected. > > Yaroslav Lobankov is working on improvement for tempest config generator > in Rally. Last time when we launch full tempest run[2], we got 1154 success > tests and only 24 skipped. Also, there is a patch, which adds x-fail > mechanism(it based on subunit-filter): you can transmit a file with test > names + reasons and rally will modify results. > > [1] - https://review.openstack.org/#/c/94473/ > > [2] - > http://logs.openstack.org/49/242849/8/check/gate-rally-dsvm-verify/e91992e/rally-verify/7_verify_results_--html.html.gz > > On Thu, Nov 26, 2015 at 1:52 PM, Ken'ichi Ohmichi <ken1ohmi...@gmail.com> > wrote: > >> Hi Daniel, >> >> Thanks for pointing this up. >> >> 2015-11-25 1:40 GMT+09:00 Daniel Mellado <daniel.mellado...@ieee.org>: >> > Hi All, >> > >> > As you might already know, within Red Hat's tempest fork, we do have one >> > tempest configuration script which was built in the past by David Kranz >> [1] >> > and that's been actively used in our CI system. Regarding this topic, >> I'm >> > aware that quite some effort has been done in the past [2] and I would >> like >> > to complete the implementation of this blueprint/spec. >> > >> > My plan would be to have this script under the /tempest/cmd or >> > /tempest/tools folder from tempest so it can be used to configure not >> the >> > tempest gate but any cloud we'd like to run tempest against. >> > >> > Adding the configuration script was discussed briefly at the Mitaka >> summit >> > in the QA Priorities meting [3]. I propose we use the existing etherpad >> to >> > continue the discussion around and tracking of implementing "tempest >> > config-create" using the downstream config script as a starting point. >> [4] >> > >> > If you have any questions, comments or opinion, please let me know. >> >> This topic have happened several times, and I also felt this kind of >> tool was very useful for Tempest users, because Tempest contains 296 >> options($ grep cfg * -R | grep Opt | wc -l) now and it is difficult to >> set the configuration up. >> However, there is a big concern: >> If the script contain a bug and creates the configuration which makes >> most tests skipped, we cannot do enough tests on the gate. >> Tempest contains 1432 tests and difficult to detect which tests are >> skipped as unexpected. >> Actually we faced unexpected skipped tests on the gate before due to >> some bug, then the problem has been fixed. >> But I can imagine this kind of problem happens after implementing this >> kind of script. >> >> So now I am feeling Tempest users need to know what cloud they want to >> test with Tempest, and need to know what tests run with Tempest. >> Testers need to know what test target/items they are testing basically. >> >> Thanks >> Ken Ohmichi >> >> --- >> >> > --- >> > [1] >> > >> https://github.com/redhat-openstack/tempest/blob/master/tools/config_tempest.py >> > [2] >> https://blueprints.launchpad.net/tempest/+spec/tempest-config-generator >> > [3] https://etherpad.openstack.org/p/mitaka-qa-priorities >> > [4] https://etherpad.openstack.org/p/tempest-cli-improvements >> > >> > >> https://github.com/openstack/qa-specs/blob/master/specs/tempest/tempest-cli-improvements.rst >> > >> > >> __________________________________________________________________________ >> > OpenStack Development Mailing List (not for usage questions) >> > Unsubscribe: >> openstack-dev-requ...@lists.openstack.org?subject:unsubscribe >> > http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev >> > >> >> __________________________________________________________________________ >> OpenStack Development Mailing List (not for usage questions) >> Unsubscribe: >> openstack-dev-requ...@lists.openstack.org?subject:unsubscribe >> http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev >> > > > > -- > Best regards, > Andrey Kurilin. > > __________________________________________________________________________ > OpenStack Development Mailing List (not for usage questions) > Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe > http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev > >
__________________________________________________________________________ OpenStack Development Mailing List (not for usage questions) Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
