I still do think that even if there are some issues addressed to the feature, such as skipping tests in the gate, the feature itself it's still good -we just won't use it for the gates- Instead it'd be used as a wrapper for a user who would be interested on trying it against a real/reals clouds.
Ken, do you really think a tempest user should know all tempest options? As you pointed out there are quite a few of them and even if they should at least know their environment, this script would set a minimum acceptable default. Do you think PTL and Pre-PTL concerns that we spoke of would still apply to that scenario? Andrey, Yaroslav. Would you like to revisit the blueprint to adapt it to tempest-cli improvements? What do you think about this, Masayuki? Thanks for all your feedback! ;) El 27/11/15 a las 00:15, Andrey Kurilin escribió: > Sorry for wrong numbers. The bug-fix for issue with counters is merged. > Correct numbers(latest result from rally's gate[1]): > - total number of executed tests: 1689 > - success: 1155 > - skipped: 534 (neutron,heat,sahara,ceilometer are disabled. [2] > should enable them) > - failed: 0 > > [1] - > http://logs.openstack.org/27/246627/11/gate/gate-rally-dsvm-verify-full/800bad0/rally-verify/7_verify_results_--html.html.gz > [2] - https://review.openstack.org/#/c/250540/ > > On Thu, Nov 26, 2015 at 3:23 PM, Yaroslav Lobankov > <yloban...@mirantis.com <mailto:yloban...@mirantis.com>> wrote: > > Hello everyone, > > Yes, I am working on this now. We have some success already, but > there is a lot of work to do. Of course, some things don't work > ideally. For example, in [2] from the previous letter we have not > 24 skipped tests, actually much more. So we have a bug somewhere :) > > Regards, > Yaroslav Lobankov. > > On Thu, Nov 26, 2015 at 3:59 PM, Andrey Kurilin > <akuri...@mirantis.com <mailto:akuri...@mirantis.com>> wrote: > > Hi! > Boris P. and I tried to push a spec[1] for automation tempest > config generator, but we did not succeed to merge it. Imo, > qa-team doesn't want to have such tool:( > > >However, there is a big concern: > >If the script contain a bug and creates the configuration > which makes > >most tests skipped, we cannot do enough tests on the gate. > >Tempest contains 1432 tests and difficult to detect which > tests are > >skipped as unexpected. > > Yaroslav Lobankov is working on improvement for tempest config > generator in Rally. Last time when we launch full tempest > run[2], we got 1154 success tests and only 24 skipped. Also, > there is a patch, which adds x-fail mechanism(it based on > subunit-filter): you can transmit a file with test names + > reasons and rally will modify results. > > [1] - https://review.openstack.org/#/c/94473/ > > [2] - > > http://logs.openstack.org/49/242849/8/check/gate-rally-dsvm-verify/e91992e/rally-verify/7_verify_results_--html.html.gz > > On Thu, Nov 26, 2015 at 1:52 PM, Ken'ichi Ohmichi > <ken1ohmi...@gmail.com <mailto:ken1ohmi...@gmail.com>> wrote: > > Hi Daniel, > > Thanks for pointing this up. > > 2015-11-25 1:40 GMT+09:00 Daniel Mellado > <daniel.mellado...@ieee.org > <mailto:daniel.mellado...@ieee.org>>: > > Hi All, > > > > As you might already know, within Red Hat's tempest > fork, we do have one > > tempest configuration script which was built in the past > by David Kranz [1] > > and that's been actively used in our CI system. > Regarding this topic, I'm > > aware that quite some effort has been done in the past > [2] and I would like > > to complete the implementation of this blueprint/spec. > > > > My plan would be to have this script under the > /tempest/cmd or > > /tempest/tools folder from tempest so it can be used to > configure not the > > tempest gate but any cloud we'd like to run tempest against. > > > > Adding the configuration script was discussed briefly at > the Mitaka summit > > in the QA Priorities meting [3]. I propose we use the > existing etherpad to > > continue the discussion around and tracking of > implementing "tempest > > config-create" using the downstream config script as a > starting point. [4] > > > > If you have any questions, comments or opinion, please > let me know. > > This topic have happened several times, and I also felt > this kind of > tool was very useful for Tempest users, because Tempest > contains 296 > options($ grep cfg * -R | grep Opt | wc -l) now and it is > difficult to > set the configuration up. > However, there is a big concern: > If the script contain a bug and creates the configuration > which makes > most tests skipped, we cannot do enough tests on the gate. > Tempest contains 1432 tests and difficult to detect which > tests are > skipped as unexpected. > Actually we faced unexpected skipped tests on the gate > before due to > some bug, then the problem has been fixed. > But I can imagine this kind of problem happens after > implementing this > kind of script. > > So now I am feeling Tempest users need to know what cloud > they want to > test with Tempest, and need to know what tests run with > Tempest. > Testers need to know what test target/items they are > testing basically. > > Thanks > Ken Ohmichi > > --- > > > --- > > [1] > > > > https://github.com/redhat-openstack/tempest/blob/master/tools/config_tempest.py > > [2] > > https://blueprints.launchpad.net/tempest/+spec/tempest-config-generator > > [3] https://etherpad.openstack.org/p/mitaka-qa-priorities > > [4] > https://etherpad.openstack.org/p/tempest-cli-improvements > > > > > > https://github.com/openstack/qa-specs/blob/master/specs/tempest/tempest-cli-improvements.rst > > > > > > __________________________________________________________________________ > > OpenStack Development Mailing List (not for usage questions) > > Unsubscribe: > openstack-dev-requ...@lists.openstack.org?subject:unsubscribe > > <http://openstack-dev-requ...@lists.openstack.org?subject:unsubscribe> > > > http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev > > > > > __________________________________________________________________________ > OpenStack Development Mailing List (not for usage questions) > Unsubscribe: > openstack-dev-requ...@lists.openstack.org?subject:unsubscribe > > <http://openstack-dev-requ...@lists.openstack.org?subject:unsubscribe> > http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev > > > > > -- > Best regards, > Andrey Kurilin. > > > __________________________________________________________________________ > OpenStack Development Mailing List (not for usage questions) > Unsubscribe: > openstack-dev-requ...@lists.openstack.org?subject:unsubscribe > <http://openstack-dev-requ...@lists.openstack.org?subject:unsubscribe> > http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev > > > > __________________________________________________________________________ > OpenStack Development Mailing List (not for usage questions) > Unsubscribe: > openstack-dev-requ...@lists.openstack.org?subject:unsubscribe > <http://openstack-dev-requ...@lists.openstack.org?subject:unsubscribe> > http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev > > > > > -- > Best regards, > Andrey Kurilin. > > > __________________________________________________________________________ > OpenStack Development Mailing List (not for usage questions) > Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe > http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
__________________________________________________________________________ OpenStack Development Mailing List (not for usage questions) Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
