On Fri, 4 Apr 2014 12:49:45 -0700 (PDT), jmaher wrote: >> overburdened in other ways (e.g., reviews). the burden >> needs to be placed on the regressing change rather than the original >> author of the test. > > I am open to ideas to help figure out the offending changes. My > understanding is many of the test failures are due to small > adjustments to the system or even the order the tests are run > such that it causes the test to fail intermittently.
On Fri, 4 Apr 2014 13:19:48 -0700, Gavin Sharp wrote: > On Fri, Apr 4, 2014 at 12:12 PM, L. David Baron <dba...@dbaron.org> wrote: >> In the latter case, the burden should fall on the regressing patch, >> and the regressing patch should get backed out instead of disabling >> the test. > > The majority of the time identifying the regressing patch is > difficult, and something that requires help from someone who can debug > the test. Hence step 1). Steps 2) and 3) are an escalation path that > might result in disabling the test or fixing it, but the primary > purpose is to ensure that decision is reached with full visibility > from the relevant owners/developers. Often enough reasonable effort does identify the cause of the regression. For tests failing more than 20% of the time, it should not be too difficult to identify the regressing push through retriggers. The key question is who bears this responsibility? David points out that authors and module owners can't keep up with this. How much investigation is required before the responsibility passes to the author or owner? I'd like to see this addressed in a policy. Sometimes, even after the regressing changes are identified, there is no response from the author or reviewer of the regressing changes. A back out is the right action here, but this is not always what happens. A policy is important to ensure that there is some path though even when there is no response from the author of the regressing change, and then no response from the test author or owner. _______________________________________________ dev-platform mailing list dev-platform@lists.mozilla.org https://lists.mozilla.org/listinfo/dev-platform
