I realised I hadn't replied to this one! > -----Original Message----- > Just so I understand the suggestion, you're saying you can make > smokestack test the changes stacked in the same sequence they're > being started by zuul, watch zuul's status mirroring the various > change ejections/test restarts... and finish testing on a given set > of scenarios before zuul does?
Many of the changes included by Zuul aren't tested, or don't need to be tested by SmokeStack - the obvious example being devstack, but there are several other projects that are tested by zuul that SS doesn't track. The changes are also primarily tested in isolation - so not in the strict ordering included in the gate. I know that with SmokeStack not following this strict ordering there is a risk that a false pass will slip through - but false fails shouldn't occur because SmokeStack does also consider the dependencies that determine the gate ordering. As such, my view is that SmokeStack could give a useful -2 vote when it knows a patch is broken - and yes, these tend to (although not 100%) complete before the gating jobs have completed for a change. > You had mentioned something about smokestack adding a -2 when it > begins testing and removing it on success, but if it doesn't > complete before zuul is ready to merge those commits then how do we > differentiate between smokestack still running (so we know to pause > the queue) and smokestack leaving that -2 indefinitely because the > test failed? Also worth noting, gerrit doesn't clear a -2 on a new > patchset upload like it does with a -1 or a positive vote, so the > proposed solution would additionally need to take care of rescinding > those -2 votes itself when new patchsets arrive to correct the > failure. I hadn't understood the implications properly - and I agree now that posting a -2 at start-of-tests would not be a good idea. My suggestion now is, in a nutshell, to allow SmokeStack to post a -2 review on a gating job so that, if the tests fail (i.e. not packaging) and it completes before the gate, then Smokestack can prevent a broken change being merged. Is that something which could work? Bob _______________________________________________ OpenStack-Infra mailing list OpenStack-Infra@lists.openstack.org http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-infra
