Hi! Boris P. and I tried to push a spec[1] for automation tempest config generator, but we did not succeed to merge it. Imo, qa-team doesn't want to have such tool:(
>However, there is a big concern: >If the script contain a bug and creates the configuration which makes >most tests skipped, we cannot do enough tests on the gate. >Tempest contains 1432 tests and difficult to detect which tests are >skipped as unexpected. Yaroslav Lobankov is working on improvement for tempest config generator in Rally. Last time when we launch full tempest run[2], we got 1154 success tests and only 24 skipped. Also, there is a patch, which adds x-fail mechanism(it based on subunit-filter): you can transmit a file with test names + reasons and rally will modify results. [1] - https://review.openstack.org/#/c/94473/ [2] - http://logs.openstack.org/49/242849/8/check/gate-rally-dsvm-verify/e91992e/rally-verify/7_verify_results_--html.html.gz On Thu, Nov 26, 2015 at 1:52 PM, Ken'ichi Ohmichi <ken1ohmi...@gmail.com> wrote: > Hi Daniel, > > Thanks for pointing this up. > > 2015-11-25 1:40 GMT+09:00 Daniel Mellado <daniel.mellado...@ieee.org>: > > Hi All, > > > > As you might already know, within Red Hat's tempest fork, we do have one > > tempest configuration script which was built in the past by David Kranz > [1] > > and that's been actively used in our CI system. Regarding this topic, I'm > > aware that quite some effort has been done in the past [2] and I would > like > > to complete the implementation of this blueprint/spec. > > > > My plan would be to have this script under the /tempest/cmd or > > /tempest/tools folder from tempest so it can be used to configure not the > > tempest gate but any cloud we'd like to run tempest against. > > > > Adding the configuration script was discussed briefly at the Mitaka > summit > > in the QA Priorities meting [3]. I propose we use the existing etherpad > to > > continue the discussion around and tracking of implementing "tempest > > config-create" using the downstream config script as a starting point. > [4] > > > > If you have any questions, comments or opinion, please let me know. > > This topic have happened several times, and I also felt this kind of > tool was very useful for Tempest users, because Tempest contains 296 > options($ grep cfg * -R | grep Opt | wc -l) now and it is difficult to > set the configuration up. > However, there is a big concern: > If the script contain a bug and creates the configuration which makes > most tests skipped, we cannot do enough tests on the gate. > Tempest contains 1432 tests and difficult to detect which tests are > skipped as unexpected. > Actually we faced unexpected skipped tests on the gate before due to > some bug, then the problem has been fixed. > But I can imagine this kind of problem happens after implementing this > kind of script. > > So now I am feeling Tempest users need to know what cloud they want to > test with Tempest, and need to know what tests run with Tempest. > Testers need to know what test target/items they are testing basically. > > Thanks > Ken Ohmichi > > --- > > > --- > > [1] > > > https://github.com/redhat-openstack/tempest/blob/master/tools/config_tempest.py > > [2] > https://blueprints.launchpad.net/tempest/+spec/tempest-config-generator > > [3] https://etherpad.openstack.org/p/mitaka-qa-priorities > > [4] https://etherpad.openstack.org/p/tempest-cli-improvements > > > > > https://github.com/openstack/qa-specs/blob/master/specs/tempest/tempest-cli-improvements.rst > > > > > __________________________________________________________________________ > > OpenStack Development Mailing List (not for usage questions) > > Unsubscribe: > openstack-dev-requ...@lists.openstack.org?subject:unsubscribe > > http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev > > > > __________________________________________________________________________ > OpenStack Development Mailing List (not for usage questions) > Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe > http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev > -- Best regards, Andrey Kurilin.
__________________________________________________________________________ OpenStack Development Mailing List (not for usage questions) Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
