On 10/5/2012 3:16 PM, Yaniv Kaul wrote:
On 10/05/2012 07:50 PM, David Kranz wrote:
A brief status update:
1. The fixes for the full tempest gate have been merged. Tempest
should run without errors except
that there is still a flakey volume test
https://bugs.launchpad.net/tempest/+bug/1056213.
This is not a bug in tempest and we are struggling with what to
do about it. Any ideas would be very welcome.
Continue executing, so people will be aware of the issue, and perhaps
be able to triage to more specific kernel versions / distributions.
(The fix proposed in
https://bugs.launchpad.net/cinder/+bug/1023755/comments/22 - to not
zero out on volume deletion, isn't the greatest from security
perspective, but that's a different matter).
2. There is now a stable/folsom branch for tempest. If you are
running tempest against a folsom cluster
you should now be using that branch of tempest.
Is there a commit policy to this branch? what about fixes from master?
We will discuss this at the next QA meeting. Obviously it has to track
stable/folsom for the other projects but
whether to backport new tests from master is a matter of resources. For
previous stable branches the
policy was that if interested parties wanted to backport fixes they could.
3. After discussion with the ci team, we are going to start gating
all core project checkins on full tempest minus flakey tests and
"non-gating" tests.
I hope this will happen next week.
What's the definition of flaky?
For this purpose, flaky means any test that should always succeed but
fails sometimes in a non-reproducible way.
For example, the volume test mentioned above has been failing perhaps
5-10% of the time. That may be acceptable for
tempest checkins, but not for other projects. Any external gating on
projects has to be rock solid.
4. We will be starting a nightly run of full tempest plus
"non-gating" tests. We will be adding stress, configuration and
performance
tests to the non-gating suite in the coming weeks.
Great to hear, I hope we'll be doing the same - lets try to coordinate
those efforts.
Indeed. There is a session in the QA track at the summit titled
"Bringing OpenStack QA into the Open":
http://summit.openstack.org/cfp/details/81
The exact purpose of this session is to encourage more open development
and execution of tests among the various OpenStack contributors.
I am hoping that people involved in testing from various companies will
attend the session.
-David
--
Mailing list: https://launchpad.net/~openstack-qa-team
Post to : openstack-qa-team@lists.launchpad.net
Unsubscribe : https://launchpad.net/~openstack-qa-team
More help : https://help.launchpad.net/ListHelp
