On Tue, Aug 12, 2014 at 1:08 PM, Doug Hellmann <d...@doughellmann.com> wrote:
> > On Aug 12, 2014, at 1:44 PM, Dolph Mathews <dolph.math...@gmail.com> > wrote: > > > On Tue, Aug 12, 2014 at 12:30 AM, Joe Gordon <joe.gord...@gmail.com> > wrote: > >> >> >> >> On Fri, Aug 8, 2014 at 6:58 AM, Kyle Mestery <mest...@mestery.com> wrote: >> >>> On Thu, Aug 7, 2014 at 1:26 PM, Joe Gordon <joe.gord...@gmail.com> >>> wrote: >>> > >>> > >>> > >>> > On Tue, Aug 5, 2014 at 9:03 AM, Thierry Carrez <thie...@openstack.org> >>> > wrote: >>> >> >>> >> Hi everyone, >>> >> >>> >> With the incredible growth of OpenStack, our development community is >>> >> facing complex challenges. How we handle those might determine the >>> >> ultimate success or failure of OpenStack. >>> >> >>> >> With this cycle we hit new limits in our processes, tools and cultural >>> >> setup. This resulted in new limiting factors on our overall velocity, >>> >> which is frustrating for developers. This resulted in the burnout of >>> key >>> >> firefighting resources. This resulted in tension between people who >>> try >>> >> to get specific work done and people who try to keep a handle on the >>> big >>> >> picture. >>> >> >>> >> It all boils down to an imbalance between strategic and tactical >>> >> contributions. At the beginning of this project, we had a strong inner >>> >> group of people dedicated to fixing all loose ends. Then a lot of >>> >> companies got interested in OpenStack and there was a surge in >>> tactical, >>> >> short-term contributions. We put on a call for more resources to be >>> >> dedicated to strategic contributions like critical bugfixing, >>> >> vulnerability management, QA, infrastructure... and that call was >>> >> answered by a lot of companies that are now key members of the >>> OpenStack >>> >> Foundation, and all was fine again. But OpenStack contributors kept on >>> >> growing, and we grew the narrowly-focused population way faster than >>> the >>> >> cross-project population. >>> >> >>> >> >>> >> At the same time, we kept on adding new projects to incubation and to >>> >> the integrated release, which is great... but the new developers you >>> get >>> >> on board with this are much more likely to be tactical than strategic >>> >> contributors. This also contributed to the imbalance. The penalty for >>> >> that imbalance is twofold: we don't have enough resources available to >>> >> solve old, known OpenStack-wide issues; but we also don't have enough >>> >> resources to identify and fix new issues. >>> >> >>> >> We have several efforts under way, like calling for new strategic >>> >> contributors, driving towards in-project functional testing, making >>> >> solving rare issues a more attractive endeavor, or hiring resources >>> >> directly at the Foundation level to help address those. But there is a >>> >> topic we haven't raised yet: should we concentrate on fixing what is >>> >> currently in the integrated release rather than adding new projects ? >>> > >>> > >>> > TL;DR: Our development model is having growing pains. until we sort >>> out the >>> > growing pains adding more projects spreads us too thin. >>> > >>> +100 >>> >>> > In addition to the issues mentioned above, with the scale of OpenStack >>> today >>> > we have many major cross project issues to address and no good place to >>> > discuss them. >>> > >>> We do have the ML, as well as the cross-project meeting every Tuesday >>> [1], but we as a project need to do a better job of actually bringing >>> up relevant issues here. >>> >>> [1] https://wiki.openstack.org/wiki/Meetings/ProjectMeeting >>> >>> >> >>> >> >>> >> We seem to be unable to address some key issues in the software we >>> >> produce, and part of it is due to strategic contributors (and core >>> >> reviewers) being overwhelmed just trying to stay afloat of what's >>> >> happening. For such projects, is it time for a pause ? Is it time to >>> >> define key cycle goals and defer everything else ? >>> > >>> > >>> > >>> > I really like this idea, as Michael and others alluded to in above, we >>> are >>> > attempting to set cycle goals for Kilo in Nova. but I think it is worth >>> > doing for all of OpenStack. We would like to make a list of key goals >>> before >>> > the summit so that we can plan our summit sessions around the goals. >>> On a >>> > really high level one way to look at this is, in Kilo we need to pay >>> down >>> > our technical debt. >>> > >>> > The slots/runway idea is somewhat separate from defining key cycle >>> goals; we >>> > can be approve blueprints based on key cycle goals without doing >>> slots. But >>> > with so many concurrent blueprints up for review at any given time, the >>> > review teams are doing a lot of multitasking and humans are not very >>> good at >>> > multitasking. Hopefully slots can help address this issue, and >>> hopefully >>> > allow us to actually merge more blueprints in a given cycle. >>> > >>> I'm not 100% sold on what the slots idea buys us. What I've seen this >>> cycle in Neutron is that we have a LOT of BPs proposed. We approve >>> them after review. And then we hit one of two issues: Slow review >>> cycles, and slow code turnaround issues. I don't think slots would >>> help this, and in fact may cause more issues. If we approve a BP and >>> give it a slot for which the eventual result is slow review and/or >>> code review turnaround, we're right back where we started. Even worse, >>> we may have not picked a BP for which the code submitter would have >>> turned around reviews faster. So we've now doubly hurt ourselves. I >>> have no idea how to solve this issue, but by over subscribing the >>> slots (e.g. over approving), we allow for the submissions with faster >>> turnaround a chance to merge quicker. With slots, we've removed this >>> capability by limiting what is even allowed to be considered for >>> review. >>> >> >> Slow review: by limiting the number of blueprints up we hope to focus our >> efforts on fewer concurrent things >> slow code turn around: when a blueprint is given a slot (runway) we will >> first make sure the author/owner is available for fast code turnaround. >> >> If a blueprint review stalls out (slow code turnaround, stalemate in >> review discussions etc.) we will take the slot and give it to another >> blueprint. >> > > How is that more efficient than today's do-the-best-we-can approach? It > just sounds like bureaucracy to me. > > Reading between the lines throughout this thread, it sounds like what > we're lacking is a reliable method to communicate review prioritization to > core reviewers. > > > It seems like this is exactly what the slots give us, though. The core > review team picks a number of slots indicating how much work they think > they can actually do (less than the available number of blueprints), and > then blueprints queue up to get a slot based on priorities and turnaround > time and other criteria that try to make slot allocation fair. By having > the slots, not only is the review priority communicated to the review team, > it is also communicated to anyone watching the project. > Slots are binary though- you're either "in" or you're "out." Review priority is a subjective grayscale, and trying to mash the two together just sounds like a unfun way to burn time. > > Doug > > > What the community wants/needs is just noise generated via IRC, email, > etherpad, launchpad, etc. It's part of a PTL's job to help filter that > noise and help communicate that back to core reviewers, but that just ends > up creating more IRC/email/etc, rather than directly impacting anyone's > experience with gerrit. > > >> >>> >>> Thanks, >>> Kyle >>> >>> >> >>> >> >>> >> On the integrated release side, "more projects" means stretching our >>> >> limited strategic resources more. Is it time for the Technical >>> Committee >>> >> to more aggressively define what is "in" and what is "out" ? If we go >>> >> through such a redefinition, shall we push currently-integrated >>> projects >>> >> that fail to match that definition out of the "integrated release" >>> inner >>> >> circle ? >>> >> >>> >> The TC discussion on what the integrated release should or should not >>> >> include has always been informally going on. Some people would like to >>> >> strictly limit to end-user-facing projects. Some others suggest that >>> >> "OpenStack" should just be about integrating/exposing/scaling smart >>> >> functionality that lives in specialized external projects, rather than >>> >> trying to outsmart those by writing our own implementation. Some >>> others >>> >> are advocates of carefully moving up the stack, and to resist from >>> >> further addressing IaaS+ services until we "complete" the pure IaaS >>> >> space in a satisfactory manner. Some others would like to build a >>> >> roadmap based on AWS services. Some others would just add anything >>> that >>> >> fits the incubation/integration requirements. >>> >> >>> >> >>> >> On one side this is a long-term discussion, but on the other we also >>> >> need to make quick decisions. With 4 incubated projects, and 2 new >>> ones >>> >> currently being proposed, there are a lot of people knocking at the >>> door. >>> > >>> > >>> > >>> > I have a slightly different short term opinion on what 'OpenStack' >>> should >>> > be: it should work really well. >>> > >>> > While we need to figure out howto increase our strategic resources, >>> > realistically I think that will still be the limiting factor in Kilo. >>> So in >>> > order to better allocate our cross project strategic resources, I >>> think we >>> > should do a project level triage ('identify projects that are likely >>> to die, >>> > regardless of what care they receive' to paraphrase the medical >>> definition >>> > of the term), and tighten our focus until we get our development >>> process >>> > into a better state. >>> > >>> > >>> >> >>> >> >>> >> Thanks for reading this braindump this far. I hope this will trigger >>> the >>> >> open discussions we need to have, as an open source project, to reach >>> >> the next level. >>> > >>> > >>> > Thank you for bringing this topic up. >>> > >>> >> >>> >> >>> >> Cheers, >>> >> >>> >> -- >>> >> Thierry Carrez (ttx) >>> >> >>> >> _______________________________________________ >>> >> OpenStack-dev mailing list >>> >> OpenStack-dev@lists.openstack.org >>> >> http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev >>> > >>> > >>> > >>> > _______________________________________________ >>> > OpenStack-dev mailing list >>> > OpenStack-dev@lists.openstack.org >>> > http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev >>> > >>> >>> _______________________________________________ >>> OpenStack-dev mailing list >>> OpenStack-dev@lists.openstack.org >>> http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev >>> >> >> >> _______________________________________________ >> OpenStack-dev mailing list >> OpenStack-dev@lists.openstack.org >> http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev >> >> > _______________________________________________ > OpenStack-dev mailing list > OpenStack-dev@lists.openstack.org > http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev > > > > _______________________________________________ > OpenStack-dev mailing list > OpenStack-dev@lists.openstack.org > http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev > >
_______________________________________________ OpenStack-dev mailing list OpenStack-dev@lists.openstack.org http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
