> This rule apparently changed four days ago, or perhaps the wiki was > wrong for a spell. I didn't get the memo that this changed; sorry I > was a bit out of date here.
Seems to have been wrong for a spell, but at least that's cleared up now. > An approval request is not two minutes of work. It involves a context > switch back to bug after it's landed. It usually involves some > back-and-forth in the bug. It involves looking up bugs to fill in the > "regression caused by" field. It involves keeping track of bugs to > make sure that the ones you care about get plus'ed. These small > context switches have an outsized cost for engineers; see [1]. I consider this us being proactive (making sure a landing makes sense) as opposed to reactive (finding unnecessary regressions weeks later). We're taking a small cost up front, and helping to walk engineers through the risk/reward process (which many are unfamiliar with). > But I also don't think you're responding to my main point, which is > that not only are uplift requests distracting, they contribute little > value, because engineers almost always get the outcome they want. You seem to be suggesting that the approval process doesn't catch/prevent mistakes. That's just not true. We still get frivolous bugs being nominated for uplift, which points to the fact that these changes would have otherwise been landed without a conversation, and possibly caused blocker regressions. We still find uplift nominations asking for unnecessary string changes late in the cycle. We still get approvals that haven't gone through a UX review. The list of things that we have an eye for goes on and on. Then there's the whole set of bugs which aren't nominated for uplift because there wasn't good enough reason to land, but which may have landed unnecessarily otherwise. > That's a question of fact, and if you think it's incorrect, we could > certainly look at the data. I always welcome data - it's a lot easier to have a conversation about than generalizations made from the experience of a single developer. Those of us in triage are in a unique position to see the forest for the trees, and that's all too often forgotten. -Alex On Apr 3, 2013, at 2:02 PM, Justin Lebar <[email protected]> wrote: > I'm sorry, Alex. My intent wasn't to pick a fight. > >>> * Constant fighting about "risk". Essentially all important patches >>> have to go through two layers of approval (review, blocking+). >>> Non-blocking patches have to go through three layers of approval >>> (review, tracking+, approval+) these days. >> >> This is false. Any bug can request approval without being tracking+ (same as >> desktop). Tracking nomination are a good way to understand whether the user >> impact is enough to warrant uplift, or to get automatic uplift in the next >> release. > > This rule apparently changed four days ago, or perhaps the wiki was > wrong for a spell. I didn't get the memo that this changed; sorry I > was a bit out of date here. > > https://wiki.mozilla.org/Release_Management/B2G_Landing?title=Release_Management%2FB2G_Landing&action=historysubmit&diff=642991&oldid=642501 > >>> In most cases, the risk assessment is completely pro-forma. Engineers >>> get what they want, in the end. But the process takes up time and >>> energy. >> >> If it's not worth the 2 minutes for an engineer to request approval, it's >> likely not worth uplifting the code change. And if approved, the bug is >> uplifted automatically by sheriffs. > > It's a bit frustrating to be told that the work I claim is distracting > is in fact not a drag on my productivity. I think I'm probably a > better judge of what is and isn't a drag than anyone else. > > An approval request is not two minutes of work. It involves a context > switch back to bug after it's landed. It usually involves some > back-and-forth in the bug. It involves looking up bugs to fill in the > "regression caused by" field. It involves keeping track of bugs to > make sure that the ones you care about get plus'ed. These small > context switches have an outsized cost for engineers; see [1]. > > But I also don't think you're responding to my main point, which is > that not only are uplift requests distracting, they contribute little > value, because engineers almost always get the outcome they want. > That's a question of fact, and if you think it's incorrect, we could > certainly look at the data. > >> You're right though - you all could lie to us about risk or reward. We're >> trusting you aren't. > > People respond to incentives. It's not a question of lying so much as > taking a position in a debate. But again, my point is that since > release drivers have to trust engineers' assessments, release drivers > contribute relatively little value to the discussion. > > I claim these approvals are a point of frustration and a drag on > productivity. I'm not suggesting that we get rid of them and leave > everything else the same; I was responding to Andreas's query as to > why branches are suboptimal even though we don't have to land on them > ourselves. I don't think we actually have a serious disagreement > here. > > [1] http://www.paulgraham.com/makersschedule.html > >> On Apr 3, 2013, at 1:27 PM, Justin Lebar <[email protected]> wrote: >> >>>>> If the past months have taught me anything, it's that "let's just >>>>> branch and double-land everything" is unfortunately not a path to >>>>> shipping quality software on time with happy, productive engineers. >>>>> So this is not a one-sided trade-off. >>>>> >>>>> I get that there are partner constraints here, and perhaps riding the >>>>> trains is incompatible with them, but I think there's a certain amount >>>>> of creativity called for here, given that what we've been doing hasn't >>>>> been working. I'd be curious to know if you have any ideas in this >>>>> respect beyond "wait a few years". >>>> >>>> I don't think there are any silver bullets I can offer. What exactly is >>>> the problem we are trying to solve here though. How often do people still >>>> have to double-land themselves? The feedback I was getting is that the >>>> vast majority of landings is done by our awesome uplifting mini-team. >>> >>> Even with our wonderful sheriff team, the many branches we have are a >>> huge pain. There are a lot of reasons. Here are some. >>> >>> * Constant fighting about "risk". Essentially all important patches >>> have to go through two layers of approval (review, blocking+). >>> Non-blocking patches have to go through three layers of approval >>> (review, tracking+, approval+) these days. >>> >>> In most cases, the risk assessment is completely pro-forma. Engineers >>> get what they want, in the end. But the process takes up time and >>> energy. I contend we would /never/ put up with this sort of process >>> BS in normal Firefox development. >>> >>> * Divergence between the branches makes landing hard. It's not at all >>> uncommon for my patches to work on trunk and cause test failures on >>> b2g18. And this problem will only get worse the longer we keep these >>> branches around. >>> >>> * Divergence between the branches causes bugs not caught by tests. >>> Again, this has happened to me personally, and has a high cost. >>> >>> * Divergence between the branches means that all branches aren't >>> tested. Essentially nobody is testing b2g nightly, so it often >>> doesn't work. So why am I even landing patches on m-c? >>> >>>> In rare instances people have to help directly (thats what we should try >>>> to reduce by >>>> keeping trunk and v1.1 close as long v1.1 is still changing). Is this >>>> accurate or do you >>>> feel that uplifting is still a pain? >>> >>> I don't think keeping v1.1 and trunk close is a goal of release >>> drivers. Their goal is to stabilize 1.1 by denying approval for risky >>> patches. That is directly at odds with keeping trunk and 1.1 close. >>> >>> If we want to keep 1.1 and trunk close, there's a simple solution: >>> Make them the same branch. If that doesn't work, perhaps we need a >>> more creative solution. But certainly what we're doing now isn't >>> working towards this goal at all. >>> >>>>> On Wed, Apr 3, 2013 at 3:59 PM, Andreas Gal <[email protected]> wrote: >>>>>> >>>>>> The vendor predicts that they will file around 300 regressions during >>>>>> the QA cycle (between now and August). I don't think we want to fix that >>>>>> on Aurora or Beta, and "lets just slip 6 weeks" is unfortunately not how >>>>>> the hardware business works. We are currently closely tied to hardware >>>>>> schedules because we are supporting individual hardware launches with >>>>>> software. The goal is over time to switch to a reference software model >>>>>> where we make a golden reference for a specific platform (a real >>>>>> hardware platform, or even just a reference platform), and OEMs make >>>>>> products out of that (similar to the Android model). Its clearly not a >>>>>> model where we have arrived at yet. For a while we will have to continue >>>>>> to be closely involved in the actual product engineering. In a year or >>>>>> two ideally we will be closer to the reference platform model, and in >>>>>> that world riding trains makes sense, and slipping will be an option. >>>>>> >>>>>> Andreas >>>>>> >>>>>> On Apr 3, 2013, at 9:41 PM, Justin Lebar <[email protected]> wrote: >>>>>> >>>>>>> This is similar to what we did with 1.0. We rode the trains until >>>>>>> beta, then we branched. >>>>>>> >>>>>>> The problem was, we branched too early. We thought the branch would >>>>>>> stabilize for a few weeks before we shipped, but in fact b2g18 has >>>>>>> been stabilizing for months and still isn't ready to ship as v1.0.1. >>>>>>> >>>>>>> To avoid making the same mistake again, I think we'd probably want to >>>>>>> say that we're not going to branch. That essentially means we need to >>>>>>> be done when Aurora branches to Beta, because changes on Beta are >>>>>>> (rightly) highly restricted. >>>>>>> >>>>>>> If we're not ready by the time we hit Beta, we slip by 6 weeks. We >>>>>>> don't put our beta users at risk for breakage to make our b2g >>>>>>> deadlines. >>>>>>> >>>>>>> Note that we'd still be double-landing most patches on aurora and m-c, >>>>>>> because we will inevitably be under a huge amount of schedule >>>>>>> pressure. But that's a heck of a lot better than double-landing on >>>>>>> b2g18 and m-c. >>>>>>> >>>>>>> On Wed, Apr 3, 2013 at 3:16 PM, Jonas Sicking <[email protected]> wrote: >>>>>>>> Hi All, >>>>>>>> >>>>>>>> It's come up a number of times that there are a fairly large number of >>>>>>>> issues with the way that we're using "v1.1 branches", i.e. b2g18 for >>>>>>>> Gecko and v1-train for gaia. >>>>>>>> >>>>>>>> I won't go in to the various issues here since I think it's generally >>>>>>>> agreed that it would be good if we could avoid them. >>>>>>>> >>>>>>>> The current release plan for v1.1 means that we have code freeze on >>>>>>>> August 12th. This is the date when, as I understand it, we absolutely >>>>>>>> have to be done with the last line of code and it's pencils down for >>>>>>>> everyone. >>>>>>>> >>>>>>>> This actually means that we would have the ability to go back to >>>>>>>> mozilla-central and gaia master right now and ride the normal release >>>>>>>> trains for the Gecko 23 release, while still reaching the release >>>>>>>> milestone for gecko before v1.1 is done. >>>>>>>> >>>>>>>> This has a lot of attractive features: >>>>>>>> * No backporting work for now! >>>>>>>> * Much simpler backporting work once we branch for aurora on may 13th. >>>>>>>> * We spend more time writing code and less time porting it. >>>>>>>> * Less risk involved when backporting patches. >>>>>>>> * We pick up a whole host of bug fixes and new features that's >>>>>>>> happened since Gecko 18 branched. Including performance work. >>>>>>>> >>>>>>>> However there are quite a few things that we need to check before we >>>>>>>> can make such a move: >>>>>>>> * Will it affect our partners ability to upgrade 1.0 users to the 1.1 >>>>>>>> release? >>>>>>>> * Are there any big and scary landings that are planned for the Gecko >>>>>>>> 23 release that we'd rather not take. Gfx layers-refactoring comes to >>>>>>>> mind. >>>>>>>> * Will we be able to take security fixes that are landing in Gecko 23 >>>>>>>> all up until August 6th? >>>>>>>> * Is mozilla-central and gaia master working well enough right now for >>>>>>>> Firefox OS that this is an option? >>>>>>>> * Will all relevant partner testing still be possible even through >>>>>>>> we'll be on mozilla-central for some of it. >>>>>>>> >>>>>>>> We also have to develop a plan for what to do if we need to fix some >>>>>>>> Gecko issue that requires large enough changes that we can't land it >>>>>>>> in the Firefox release trains. At that point we have to branch away >>>>>>>> from the normal Firefox release branch. This means that all fixes that >>>>>>>> go into the Firefox branch will have to also be ported to the B2G >>>>>>>> branch. >>>>>>>> >>>>>>>> All in all I think the advantages outweigh the disadvantages here. >>>>>>>> Things from Firefox development landing in aurora and beta tend to be >>>>>>>> very safe. Definitely safer than the sum of the large amounts of work >>>>>>>> we'll still be doing for B2G. So if we can work through the issue list >>>>>>>> above, I think we should do it. >>>>>>>> >>>>>>>> Would love input from both developers and product on this. >>>>>>>> Unfortunately I'll be on vacation thursday and friday so it would be >>>>>>>> great to get help driving looking into the checklist above during that >>>>>>>> time. >>>>>>>> >>>>>>>> / Jonas >>>>>>>> _______________________________________________ >>>>>>>> dev-b2g mailing list >>>>>>>> [email protected] >>>>>>>> https://lists.mozilla.org/listinfo/dev-b2g >>>>>>> _______________________________________________ >>>>>>> dev-b2g mailing list >>>>>>> [email protected] >>>>>>> https://lists.mozilla.org/listinfo/dev-b2g >>>>>> >>>> >>> _______________________________________________ >>> dev-b2g mailing list >>> [email protected] >>> https://lists.mozilla.org/listinfo/dev-b2g >> _______________________________________________ dev-b2g mailing list [email protected] https://lists.mozilla.org/listinfo/dev-b2g
