On Tue, Feb 10, 2015, at 05:19 AM, Thierry Carrez wrote: > Joe, Matt & Matthew: > > I hear your frustration with broken stable branches. With my > vulnerability management team member hat, responsible for landing > patches there with a strict deadline, I can certainly relate with the > frustration of having to dive in to unbork the branch in the first > place, rather than concentrate on the work you initially planned on > doing. > > That said, wearing my stable team member hat, I think it's a bit unfair > to say that things are worse than they were and call for dramatic > action. The stable branch team put a structure in place to try to > continuously fix the stable branches rather than reactively fix it when > we need it to work. Those champions have been quite active[1] unbreaking > it in the past months. I'd argue that the branch is broken much less > often than it used to. That doesn't mean it's never broken, though, or > that those people are magicians. > > One issue in the current situation is that the two groups (you and the > stable maintainers) seem to work in parallel rather than collaborate. > It's quite telling that the two groups maintained separate etherpads to > keep track of the fixes that needed landing.
I agree we should figure out a way to communicate better between the various teams involved in fixing the issues. Consolidating some of the communication channels we use should help. For example, if we settle on a single channel to use on IRC then we can log that channel so anyone can review the history of a given issue, either to catch up on what happened overnight or to understand how an issue was resolved long after the fact. At one point we created #openstack-gate for this, but I don't see many people there now and it's not being logged. Maybe we should just use #openstack-dev, since that isn't as active for other purposes now? > > [1] https://etherpad.openstack.org/p/stable-tracker > > Matthew Treinish wrote: > > So I think it's time we called the icehouse branch and marked it EOL. We > > originally conditioned the longer support window on extra people stepping > > forward to keep things working. I believe this latest issue is just the > > latest > > indication that this hasn't happened. Issue 1 listed above is being caused > > by > > the icehouse branch during upgrades. The fact that a stable release was > > pushed > > at the same time things were wedged on the juno branch is just the latest > > evidence to me that things aren't being maintained as they should be. > > Looking at > > the #openstack-qa irc log from today or the etherpad about trying to sort > > this > > issue should be an indication that no one has stepped up to help with the > > maintenance and it shows given the poor state of the branch. > > I disagree with the assessment. People have stepped up. I think the > stable branches are less often broken than they were, and stable branch > champions (as their tracking etherpad shows) have made a difference. They seem to be pretty consistently broken, at least lately, but the causes have also been consistent. This cycle we changed the way we manage libraries (dropping alpha release versions) and test tools (branchless tempest) without fully understanding the effects those changes would have throughout the complex testing systems we have in place. As the changes have rippled through the system, we've found more and more unintended consequences, some of which have required making other big changes to the way test jobs are defined and to how we manage requirements. I'm not convinced we would have identified all of those issues even if we had spent more time reasoning through the changes before making them, given the complexity. So, although it has been a frustrating period, in retrospect I think there were probably a few things we could have done differently early on but it was largely necessary to do it this way to uncover the issues we couldn't predict. I think we're approaching a good state, too. The series of patches Joe, Matt, and Sean came up with yesterday should unwedge icehouse. Then we can resume the work to cap the requirements lists, and that will avoid breaking backwards compatibility test jobs with new releases of libraries. We've also stopped testing master branches of libraries against the stable branches where we won't be running the code, so development on the master branches of those libraries should already be unblocked (although we're not releasing anything until this is sorted out, which is another sort of block). > There just has been more issues as usual recently and they probably > couldn't keep track. It's not a fun job to babysit stable branches, > belittling the stable branch champions results is not the best way to > encourage them to continue in this position. I agree that they could > work more with the QA team when they get overwhelmed, and raise more red > flags when they just can't keep up. > > I also disagree with the proposed solution. We announced a support > timeframe for Icehouse, our downstream users made plans around it, so we > should stick to it as much as we can. If we dropped stable branch > support every time a patch can't be landed there, there would just not > be any stable branch. > > Joe Gordon wrote: > > Where is it documented that Adam is the Juno branch champion and Ihar is > > Icehouse's? I didn't see it anywhere in the wiki. > > It was announced here: > http://lists.openstack.org/pipermail/openstack-dev/2014-November/050390.html > > I agree it should be documented on the wiki. > > -- > Thierry Carrez (ttx) > > __________________________________________________________________________ > OpenStack Development Mailing List (not for usage questions) > Unsubscribe: > openstack-dev-requ...@lists.openstack.org?subject:unsubscribe > http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev __________________________________________________________________________ OpenStack Development Mailing List (not for usage questions) Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
