Justin, let's take a step back and not make invalid assertions. > * Constant fighting about "risk". Essentially all important patches > have to go through two layers of approval (review, blocking+). > Non-blocking patches have to go through three layers of approval > (review, tracking+, approval+) these days.
This is false. Any bug can request approval without being tracking+ (same as desktop). Tracking nomination are a good way to understand whether the user impact is enough to warrant uplift, or to get automatic uplift in the next release. > In most cases, the risk assessment is completely pro-forma. Engineers > get what they want, in the end. But the process takes up time and > energy. If it's not worth the 2 minutes for an engineer to request approval, it's likely not worth uplifting the code change. And if approved, the bug is uplifted automatically by sheriffs. You're right though - you all could lie to us about risk or reward. We're trusting you aren't. > I contend we would /never/ put up with this sort of process > BS in normal Firefox development. This is so apples and oranges it's astonishing. How can you compare the landing requirements of desktop releases (which involve no partners, have massive pre-release testing populations, is a mature product, and has no hard feature scope) to B2G releases? -Alex On Apr 3, 2013, at 1:27 PM, Justin Lebar <[email protected]> wrote: >>> If the past months have taught me anything, it's that "let's just >>> branch and double-land everything" is unfortunately not a path to >>> shipping quality software on time with happy, productive engineers. >>> So this is not a one-sided trade-off. >>> >>> I get that there are partner constraints here, and perhaps riding the >>> trains is incompatible with them, but I think there's a certain amount >>> of creativity called for here, given that what we've been doing hasn't >>> been working. I'd be curious to know if you have any ideas in this >>> respect beyond "wait a few years". >> >> I don't think there are any silver bullets I can offer. What exactly is the >> problem we are trying to solve here though. How often do people still have >> to double-land themselves? The feedback I was getting is that the vast >> majority of landings is done by our awesome uplifting mini-team. > > Even with our wonderful sheriff team, the many branches we have are a > huge pain. There are a lot of reasons. Here are some. > > * Constant fighting about "risk". Essentially all important patches > have to go through two layers of approval (review, blocking+). > Non-blocking patches have to go through three layers of approval > (review, tracking+, approval+) these days. > > In most cases, the risk assessment is completely pro-forma. Engineers > get what they want, in the end. But the process takes up time and > energy. I contend we would /never/ put up with this sort of process > BS in normal Firefox development. > > * Divergence between the branches makes landing hard. It's not at all > uncommon for my patches to work on trunk and cause test failures on > b2g18. And this problem will only get worse the longer we keep these > branches around. > > * Divergence between the branches causes bugs not caught by tests. > Again, this has happened to me personally, and has a high cost. > > * Divergence between the branches means that all branches aren't > tested. Essentially nobody is testing b2g nightly, so it often > doesn't work. So why am I even landing patches on m-c? > >> In rare instances people have to help directly (thats what we should try to >> reduce by >> keeping trunk and v1.1 close as long v1.1 is still changing). Is this >> accurate or do you >> feel that uplifting is still a pain? > > I don't think keeping v1.1 and trunk close is a goal of release > drivers. Their goal is to stabilize 1.1 by denying approval for risky > patches. That is directly at odds with keeping trunk and 1.1 close. > > If we want to keep 1.1 and trunk close, there's a simple solution: > Make them the same branch. If that doesn't work, perhaps we need a > more creative solution. But certainly what we're doing now isn't > working towards this goal at all. > >>> On Wed, Apr 3, 2013 at 3:59 PM, Andreas Gal <[email protected]> wrote: >>>> >>>> The vendor predicts that they will file around 300 regressions during the >>>> QA cycle (between now and August). I don't think we want to fix that on >>>> Aurora or Beta, and "lets just slip 6 weeks" is unfortunately not how the >>>> hardware business works. We are currently closely tied to hardware >>>> schedules because we are supporting individual hardware launches with >>>> software. The goal is over time to switch to a reference software model >>>> where we make a golden reference for a specific platform (a real hardware >>>> platform, or even just a reference platform), and OEMs make products out >>>> of that (similar to the Android model). Its clearly not a model where we >>>> have arrived at yet. For a while we will have to continue to be closely >>>> involved in the actual product engineering. In a year or two ideally we >>>> will be closer to the reference platform model, and in that world riding >>>> trains makes sense, and slipping will be an option. >>>> >>>> Andreas >>>> >>>> On Apr 3, 2013, at 9:41 PM, Justin Lebar <[email protected]> wrote: >>>> >>>>> This is similar to what we did with 1.0. We rode the trains until >>>>> beta, then we branched. >>>>> >>>>> The problem was, we branched too early. We thought the branch would >>>>> stabilize for a few weeks before we shipped, but in fact b2g18 has >>>>> been stabilizing for months and still isn't ready to ship as v1.0.1. >>>>> >>>>> To avoid making the same mistake again, I think we'd probably want to >>>>> say that we're not going to branch. That essentially means we need to >>>>> be done when Aurora branches to Beta, because changes on Beta are >>>>> (rightly) highly restricted. >>>>> >>>>> If we're not ready by the time we hit Beta, we slip by 6 weeks. We >>>>> don't put our beta users at risk for breakage to make our b2g >>>>> deadlines. >>>>> >>>>> Note that we'd still be double-landing most patches on aurora and m-c, >>>>> because we will inevitably be under a huge amount of schedule >>>>> pressure. But that's a heck of a lot better than double-landing on >>>>> b2g18 and m-c. >>>>> >>>>> On Wed, Apr 3, 2013 at 3:16 PM, Jonas Sicking <[email protected]> wrote: >>>>>> Hi All, >>>>>> >>>>>> It's come up a number of times that there are a fairly large number of >>>>>> issues with the way that we're using "v1.1 branches", i.e. b2g18 for >>>>>> Gecko and v1-train for gaia. >>>>>> >>>>>> I won't go in to the various issues here since I think it's generally >>>>>> agreed that it would be good if we could avoid them. >>>>>> >>>>>> The current release plan for v1.1 means that we have code freeze on >>>>>> August 12th. This is the date when, as I understand it, we absolutely >>>>>> have to be done with the last line of code and it's pencils down for >>>>>> everyone. >>>>>> >>>>>> This actually means that we would have the ability to go back to >>>>>> mozilla-central and gaia master right now and ride the normal release >>>>>> trains for the Gecko 23 release, while still reaching the release >>>>>> milestone for gecko before v1.1 is done. >>>>>> >>>>>> This has a lot of attractive features: >>>>>> * No backporting work for now! >>>>>> * Much simpler backporting work once we branch for aurora on may 13th. >>>>>> * We spend more time writing code and less time porting it. >>>>>> * Less risk involved when backporting patches. >>>>>> * We pick up a whole host of bug fixes and new features that's >>>>>> happened since Gecko 18 branched. Including performance work. >>>>>> >>>>>> However there are quite a few things that we need to check before we >>>>>> can make such a move: >>>>>> * Will it affect our partners ability to upgrade 1.0 users to the 1.1 >>>>>> release? >>>>>> * Are there any big and scary landings that are planned for the Gecko >>>>>> 23 release that we'd rather not take. Gfx layers-refactoring comes to >>>>>> mind. >>>>>> * Will we be able to take security fixes that are landing in Gecko 23 >>>>>> all up until August 6th? >>>>>> * Is mozilla-central and gaia master working well enough right now for >>>>>> Firefox OS that this is an option? >>>>>> * Will all relevant partner testing still be possible even through >>>>>> we'll be on mozilla-central for some of it. >>>>>> >>>>>> We also have to develop a plan for what to do if we need to fix some >>>>>> Gecko issue that requires large enough changes that we can't land it >>>>>> in the Firefox release trains. At that point we have to branch away >>>>>> from the normal Firefox release branch. This means that all fixes that >>>>>> go into the Firefox branch will have to also be ported to the B2G >>>>>> branch. >>>>>> >>>>>> All in all I think the advantages outweigh the disadvantages here. >>>>>> Things from Firefox development landing in aurora and beta tend to be >>>>>> very safe. Definitely safer than the sum of the large amounts of work >>>>>> we'll still be doing for B2G. So if we can work through the issue list >>>>>> above, I think we should do it. >>>>>> >>>>>> Would love input from both developers and product on this. >>>>>> Unfortunately I'll be on vacation thursday and friday so it would be >>>>>> great to get help driving looking into the checklist above during that >>>>>> time. >>>>>> >>>>>> / Jonas >>>>>> _______________________________________________ >>>>>> dev-b2g mailing list >>>>>> [email protected] >>>>>> https://lists.mozilla.org/listinfo/dev-b2g >>>>> _______________________________________________ >>>>> dev-b2g mailing list >>>>> [email protected] >>>>> https://lists.mozilla.org/listinfo/dev-b2g >>>> >> > _______________________________________________ > dev-b2g mailing list > [email protected] > https://lists.mozilla.org/listinfo/dev-b2g _______________________________________________ dev-b2g mailing list [email protected] https://lists.mozilla.org/listinfo/dev-b2g
