In general I agree with what has been outline in Jim's reply below. Jim, I have a couple replies inline though :)
Bob, I think calling what we talked about a "gate" is probably the wrong word. This is third party testing running as a pre-gate check which we are trying to make a bit more formal. Can you prevent bad code from landing by running tests as soon as someone prevents a branch: Yes!. Can you gate everything and guarantee nothing slips in: No. Is it worth doing? I think so because at the end of the day preventing bad code from landing is always good. Much of the SmokeStack improvements we talked about (like isolating package build failures from torpedo/test failures) is going to happen anyway. So what we end up with is basically something where we can confidently at pre-gate time run tests and provide feedback in the form of a -1 in the verified column without someone like me having to approve it. I understand you'd like that to be a -2 if the tests fail... but is automatically posting a -1 good enough for now. ----- Original Message ----- > From: "James E. Blair" <jebl...@openstack.org> > To: "Bob Ball" <bob.b...@citrix.com> > Cc: openstack-infra@lists.openstack.org > Sent: Friday, August 9, 2013 11:56:28 AM > Subject: Re: [OpenStack-Infra] Smokestack posting +/-2 votes > > Bob Ball <bob.b...@citrix.com> writes: > > > What are the thoughts on how an external system can add gating > > comments? > > This is very unlikely to happen. > > Multiple gating systems for one project (or set of projects in our case) > will not work. Zuul does quite a bit of work to try to merge changes as > quickly as possible, including speculative execution (where it builds > git trees for several projects that represent the expected repo states > in the future after many changes merge). It also recognizes updates to > the current state of changes in review, determining whether they can > merge based on current review criteria, dependencies, etc. > > In other words, Zuul needs a global and authoritative view of what is > happening. If there is an outside influence that may cause an > unpredictable change (such as a random -2 showing up), Zuul will not be > able to make the correct choices. > > There is another important issue -- we can not gate on something that is > not run by the infrastructure team. We have a responsibility to the > developers of the project to keep this system running. If something > breaks, any one of several people on the (open; anyone is welcome!) > infrastructure team needs to be able to fix it. If we started running > gate tests on an external system and it stopped working, development on > the entire project would stop, and we would be powerless to fix it. > > On a constructive note, much of what smokestack does could be > incorporated into the current infrastructure. The "devstack" jenkins > nodes are becoming increasingly inaccurately named, as they are already > running tests that have nothing to do with devstack. Think of them as > general purpose single-use jenkins slaves. It would not be too > difficult to have a jenkins job build packages, and then have further > jenkins jobs use those packages to stand up a multi-node openstack (if > you need three nodes, write three jobs that each run on a single-use > full-access slave). Obviously this would be quite resource intensive, > but I believe it's possible. Obviously I'm very interested in this idea. Using packages and configuration management in multi-node environments to test upstream is really the reason SmokeStack exists. Once upon a time we did have some Jenkins jobs that used packages and once devstack came around interest just wasn't there for us to maintain those (in the upstream openstack Jenkins even). If things had gone differently... well lets just say there may not be a SmokeStack around today. Perhaps that job was before its time though and at the time the community seem violently against using packages of any sort in upstream. Two years later: if there is an open door here I'd like to pursue it... it is a lot of work though and I certainly think there is a good bit to learn from the workflow we use in SmokeStack now. In the meantime there is still a good bit of ground to hold with SmokeStack which isn't easily swapped out. So this is encouraging... and we have lots of work to do. :) > > The problem, as I understand it, is that we can't run Xen nested on any > of our current cloud providers. If there is no way to do that, then I > believe we would need a new source of test resources for this. I think > we talked about this a bit at the summit, but I think in general if > someone wants to provide new testing resources, they would need to be > sufficient to support our resource usage (which is large and growing), > supported by a real ops team (because we aren't one) with something like > an SLA. We discussed some ideas around bare metal testing at the summit > here: https://etherpad.openstack.org/havana-bare-metal-testing > > Even without gating, the advisory reporting from smokestack is hugely > valuable. I think any increase in reliability and speed (such as > automating the failure detection as you were talking about) will be > perceived by the review community and they will act accordingly, giving > it significant weight in their reviews. Exactly. > > -Jim > > _______________________________________________ > OpenStack-Infra mailing list > OpenStack-Infra@lists.openstack.org > http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-infra > _______________________________________________ OpenStack-Infra mailing list OpenStack-Infra@lists.openstack.org http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-infra
