In general I'm in favour of this proposal, although it will probably come back to haunt me in the not-too-distant future. That being said I would like to know what criteria you used to distinguish "reliable" talos tests from the rest.
kats On Fri, Aug 14, 2015 at 5:26 PM, Vladan Djeric <vdje...@mozilla.com> wrote: > The perf team and the A-Team would like to test out a new policy: we want > to back out patches that cause significant Talos regressions on Windows > builds. We would like to get developers’ feedback before starting this > experiment. > > Why are we doing this? > > Essentially, we would like more Talos regressions to get fixed and Firefox > performance to improve. We want to test a 48-hour backout policy because we > noticed that patch authors often don't fix regressions quickly. If a > regression sits in the tree for days, it becomes difficult to back out, and > it becomes much more likely that the regression will end up riding the > trains to Release by default. > > Note that we already have a policy of backing out regressions of 3% or more > if the patch author does not respond at all on the regression bug for 3 > business days. See > https://wiki.mozilla.org/Buildbot/Talos/RegressionBugsHandling > > The new policy is more aggressive. We think a patch that regresses > performance significantly should be backed out quickly, and re-landed when > its performance is acceptable. > > How will the backouts work? > > The A-Team perf sheriffs will create a Talos regression bug as soon as a > regression is confirmed using Talos re-triggers. The patch author and > reviewer will be CC’ed, and if they don’t provide an explanation for why > the regression is acceptable, the patch will be backed out. The goal is to > back out unjustified regressions within 48 hours of landing. We’d like to > give the patch author about 24 hours to reply after the regression bug is > filed. > > The A-Team has been working hard on improving the tools for understanding > Talos regressions (e.g. Perfherder), and we think debugging a Talos > regression is a much less painful process these days. For example, there is > now a highly usable view to visualize the comparison between a proposed > patch against a baseline revision at > https://treeherder.mozilla.org/perf.html#/comparechooser > > Will all regressions get backed out? > > No. Only regressions of 10% or more, on reliable Talos tests, on Windows, > will face automatic backouts during this trial period. Given historical > trends, we are anticipating about one backout per week. > > > List of reliable Talos tests: ts_paint, sessionrestore, tp5o_scroll, > tscrollx, tresize, TART, tsvgx, tp5o > > How are you testing this new policy? > > First off, we want developers’ feedback before trialing this new policy. > > I would like us to collect feedback and then start enforcing the new policy > sometime next week. You can talk to us on the newsgroups or on #perf. You > can also reach me directly (my IRC nick is “vladan”). > > We’ll post our conclusions on the newsgroups after a couple of months of > enforcing the policy, and then we can all re-evaluate the backout policy > together. > > Who will be the perf sheriffs? > > Joel Maher, Vaibhav Agrawal > _______________________________________________ > dev-platform mailing list > dev-platform@lists.mozilla.org > https://lists.mozilla.org/listinfo/dev-platform _______________________________________________ dev-platform mailing list dev-platform@lists.mozilla.org https://lists.mozilla.org/listinfo/dev-platform
