Over the last few weeks I've been talking to Dan Prince about what we might be able to do with SmokeStack and whether we can get it to a stage where it can veto a change that it knows has broken tests.
The first part of this would need to be SmokeStack differentiating between the types of failure - this should be relatively easy and is planned in the next few weeks. What we'd like to see once we have implemented this is a workflow similar to https://docs.google.com/presentation/d/1qTvZ8kmByOrtilYhc_BiwOhkzpN-GT0UTbniusnKohI/edit?usp=sharing, with appropriate integration with the gate. What are the thoughts on how an external system can add gating comments? My initial thought is that as soon as SmokeStack picks up a job it could -2 comment, preventing any merge. Then it will be responsible for either a final +2 comment if the tests pass, -2 comment if a test explicitly fails and a 0 comment if a packaging failure or unknown failure occurs. In the general case this should work fine because SmokeStack is running at sub 30 minutes per change, so Jenkins (typically taking around an hour) should complete well after SmokeStack and therefore would do the right thing without any further changes. What are the thoughts on the case where SmokeStack takes longer to respond than Jenkins? Some options might be: * Merge change after SmokeStack has approved (small risk that the merge might fail if SmokeStack approved changes in a different order) * Re-submit to the merge jobs after Smokestack has completed (ultra-safe, but with a double test for the happy path) * Merge anyway and accept that SmokeStack might find things to be broken in this and other changes In the final "merge anyway" case, we may need some way for users to ignore_smokestack -2's as a previous change may have broken tests not checked by the Jenkins gate. We'd need SmokeStack to have a retest keyword it watches for as well - or possibly just act on the recheck/reverify commands currently used. Thanks, Bob _______________________________________________ OpenStack-Infra mailing list OpenStack-Infra@lists.openstack.org http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-infra
