On 08/14/2013 06:08 AM, Bob Ball wrote: > I realised I hadn't replied to this one! > >> -----Original Message----- Just so I understand the suggestion, >> you're saying you can make smokestack test the changes stacked in >> the same sequence they're being started by zuul, watch zuul's >> status mirroring the various change ejections/test restarts... and >> finish testing on a given set of scenarios before zuul does? > > Many of the changes included by Zuul aren't tested, or don't need to > be tested by SmokeStack - the obvious example being devstack, but > there are several other projects that are tested by zuul that SS > doesn't track. > > The changes are also primarily tested in isolation - so not in the > strict ordering included in the gate. I know that with SmokeStack > not following this strict ordering there is a risk that a false pass > will slip through - but false fails shouldn't occur because > SmokeStack does also consider the dependencies that determine the > gate ordering.
If you aren't testing with the re-ordering, then you are doing the equivalent of testing the patch as uploaded. (which, it turns out, is super useful) > As such, my view is that SmokeStack could give a useful -2 vote when > it knows a patch is broken - and yes, these tend to (although not > 100%) complete before the gating jobs have completed for a change. > >> You had mentioned something about smokestack adding a -2 when it >> begins testing and removing it on success, but if it doesn't >> complete before zuul is ready to merge those commits then how do >> we differentiate between smokestack still running (so we know to >> pause the queue) and smokestack leaving that -2 indefinitely >> because the test failed? Also worth noting, gerrit doesn't clear a >> -2 on a new patchset upload like it does with a -1 or a positive >> vote, so the proposed solution would additionally need to take care >> of rescinding those -2 votes itself when new patchsets arrive to >> correct the failure. > > I hadn't understood the implications properly - and I agree now that > posting a -2 at start-of-tests would not be a good idea. > > My suggestion now is, in a nutshell, to allow SmokeStack to post a -2 > review on a gating job so that, if the tests fail (i.e. not > packaging) and it completes before the gate, then Smokestack can > prevent a broken change being merged. > > Is that something which could work? In practice, honestly, until I see it run with automatic -1's for a while, I think we're stepping several steps beyond what we know. I want to suggest that: a) get auto -1 to a place where you're happy with it b) after that - let's look at stats on incidences of -1 votes from smokestack that were overridden erroneously. I think both of the above are useful and non-controversial things. And then once they're done, we'll have better data to base further discussions on. _______________________________________________ OpenStack-Infra mailing list OpenStack-Infra@lists.openstack.org http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-infra
