Jim, we discussed this at the QA meeting and would appreciate your going
ahead with this.
We are going to raise the current problems at the nova meeting today and
see if we can get more attention to the
breakages whose tests in tempest have been commented out. We hope that
(3), because it directly ties a failure to a
particular commit, will make it easier for core team developers to fix
problems quickly when they
arise.
We would really like to run full tempest on all projects. Running
tempest in parallel is blocked due to
a problem with the nose multiprocessing plugin. No one on the qa team
seems to know how to fix that.
If you have any ideas I think Jay is the closest to that issue.
-David
On 9/5/2012 9:03 PM, James E. Blair wrote:
Hi,
In last weeks QA meeting, there was some talk about making improvements
around devstack-gate and tempest.
As I understand it, one of the problems is that things are changing in
other projects that are causing tests in the full (non-smoke) tempest
suite to fail, which prevents changes being merged to tempest. This is
unique to tempest because of the asymmetrical gate currently in place,
where tempest is gated on the full suite but other projects are gated
only on the smoke tests.
There was talk about solving this by adding a second, non-gating, run of
tempest to run those jobs. I'd like to keep the number of
devstack/tempest runs low, because of the heavy resource consumption --
especially when you consider that we've committed to doing cross-version
testing, which will start to exponentially increase the number of
devstack runs we do.
Here's what I'd like to propose instead:
1) Make the tempest gate symmetrical -- cross-project gating works best
when all the projects gate on the same tests so that one project can't
break another.
2) Add more tests to the set of tests that get run in the
devstack/tempest gate (currently the smoke tests). It sounds like
several tests in the full suite really are breaking because of changes
in other projects -- those should probably be considered for promotion.
3) Modify devstack-gate to run tempest a second time (on the same host,
after completing the smoke tests), to run the rest of the suite, but
ignore the results of that for gating purposes. This way the results
will be available for every change, without affecting gating or
consuming more virtual machines. In other words, have devstack gate:
run devstack
run smoke tests
save return code
run full suite - smoke tests
exit with saved return code
How does that sound?
-Jim
--
Mailing list: https://launchpad.net/~openstack-qa-team
Post to : openstack-qa-team@lists.launchpad.net
Unsubscribe : https://launchpad.net/~openstack-qa-team
More help : https://help.launchpad.net/ListHelp
