Tests are reliable if they detect regressions, aren't very noisy, and if they measure things that have a real impact on actual Firefox user experience.
We're using past experience with the tests to determine which ones meet this criteria. On Friday, August 14, 2015 at 5:45:35 PM UTC-4, Kartikaya Gupta wrote: > In general I'm in favour of this proposal, although it will probably > come back to haunt me in the not-too-distant future. That being said I > would like to know what criteria you used to distinguish "reliable" > talos tests from the rest. > > kats > > On Fri, Aug 14, 2015 at 5:26 PM, Vladan Djeric <vdje...@mozilla.com> wrote: > > The perf team and the A-Team would like to test out a new policy: we want > > to back out patches that cause significant Talos regressions on Windows > > builds. We would like to get developers' feedback before starting this > > experiment. > > > > Why are we doing this? > > > > Essentially, we would like more Talos regressions to get fixed and Firefox > > performance to improve. We want to test a 48-hour backout policy because we > > noticed that patch authors often don't fix regressions quickly. If a > > regression sits in the tree for days, it becomes difficult to back out, and > > it becomes much more likely that the regression will end up riding the > > trains to Release by default. > > > > Note that we already have a policy of backing out regressions of 3% or more > > if the patch author does not respond at all on the regression bug for 3 > > business days. See > > https://wiki.mozilla.org/Buildbot/Talos/RegressionBugsHandling > > > > The new policy is more aggressive. We think a patch that regresses > > performance significantly should be backed out quickly, and re-landed when > > its performance is acceptable. > > > > How will the backouts work? > > > > The A-Team perf sheriffs will create a Talos regression bug as soon as a > > regression is confirmed using Talos re-triggers. The patch author and > > reviewer will be CC'ed, and if they don't provide an explanation for why > > the regression is acceptable, the patch will be backed out. The goal is to > > back out unjustified regressions within 48 hours of landing. We'd like to > > give the patch author about 24 hours to reply after the regression bug is > > filed. > > > > The A-Team has been working hard on improving the tools for understanding > > Talos regressions (e.g. Perfherder), and we think debugging a Talos > > regression is a much less painful process these days. For example, there is > > now a highly usable view to visualize the comparison between a proposed > > patch against a baseline revision at > > https://treeherder.mozilla.org/perf.html#/comparechooser > > > > Will all regressions get backed out? > > > > No. Only regressions of 10% or more, on reliable Talos tests, on Windows, > > will face automatic backouts during this trial period. Given historical > > trends, we are anticipating about one backout per week. > > > > > > List of reliable Talos tests: ts_paint, sessionrestore, tp5o_scroll, > > tscrollx, tresize, TART, tsvgx, tp5o > > > > How are you testing this new policy? > > > > First off, we want developers' feedback before trialing this new policy. > > > > I would like us to collect feedback and then start enforcing the new policy > > sometime next week. You can talk to us on the newsgroups or on #perf. You > > can also reach me directly (my IRC nick is "vladan"). > > > > We'll post our conclusions on the newsgroups after a couple of months of > > enforcing the policy, and then we can all re-evaluate the backout policy > > together. > > > > Who will be the perf sheriffs? > > > > Joel Maher, Vaibhav Agrawal > > _______________________________________________ > > dev-platform mailing list > > dev-platform@lists.mozilla.org > > https://lists.mozilla.org/listinfo/dev-platform _______________________________________________ dev-platform mailing list dev-platform@lists.mozilla.org https://lists.mozilla.org/listinfo/dev-platform
