On Fri, Apr 4, 2014 at 12:12 PM, L. David Baron <dba...@dbaron.org> wrote: >> Escalation path: >> 1) Ensure we have a bug on file, with the test author, reviewer, module >> owner, and any other interested parties, links to logs, etc. >> 2) We need to needinfo? and expect a response within 2 business days, this >> should be clear in a comment. >> 3) In the case we don't get a response, request a needinfo? from the module >> owner >> with the expectation of 2 days for a response and getting someone to take >> action. >> 4) In the case we go another 2 days with no response from a module owner, we >> will disable the test.
> In the latter case, the burden should fall on the regressing patch, > and the regressing patch should get backed out instead of disabling > the test. The majority of the time identifying the regressing patch is difficult, and something that requires help from someone who can debug the test. Hence step 1). Steps 2) and 3) are an escalation path that might result in disabling the test or fixing it, but the primary purpose is to ensure that decision is reached with full visibility from the relevant owners/developers. > If this plan is applied to existing tests, then it will lead to > style system mochitests being turned off due to other regressions > because I'm the person who wrote them and the module owner, and I > don't always have time to deal with regressions in other parts of > code (e.g., the JS engine) leading to these tests failing > intermittently. That might be the right call. That decision needs to not be reached only by sheriffs, which is occasionally the case now. This proposed policy helps ensure that the module owner has at least an opportunity to influence the decision about how to proceed with tests in their module. If a style system test is failing intermittently enough that it's a significant burden on sheriffs, and you can't find someone to look into it in a reasonable time frame, then disabling it is probably the only option. The judgement call about "intermittently enough" and the relative tradeoff of sheriff burden vs. test coverage is one that needs to be made by a combination of you and sheriffs, and that's really all that this policy is meant to enforce. Gavin _______________________________________________ dev-platform mailing list dev-platform@lists.mozilla.org https://lists.mozilla.org/listinfo/dev-platform
