On Friday 2014-04-04 12:49 -0700, jmaher wrote: > > If this plan is applied to existing tests, then it will lead to > > style system mochitests being turned off due to other regressions > > because I'm the person who wrote them and the module owner, and I > > don't always have time to deal with regressions in other parts of > > code (e.g., the JS engine) leading to these tests failing > > intermittently. > > > > If that happens, we won't have the test coverage we need to add new > > CSS properties or values. > > Interesting point. Are these tests failing often? Can we invest some > minimal time into these to make them more reliable from a test case or test > harness perspective?
They're not failing often. But there has been at least one (and I think more) occasion where changes in very unrelated code (one was tracked down to a JS JIT change) caused them, and a bunch of other tests, to start failing intermittently at higher frequency. I think detecting this sort of pattern, of a low level change causing large numbers of tests to fail at moderate frequency, should be detected in a way that doesn't lead to us disabling large numbers of tests (and then probably failing to re-enable them when the problem is resolved). I think we could use better tools for finding regression windows for intermittent failures. I'd also note that I've been trying to give frequent intermittent oranges in layout a reasonably high priority, and encourage others to do the same. I think we've been doing a reasonably good job at this lately, although rates of progress have varied. -David -- 𝄞 L. David Baron http://dbaron.org/ 𝄂 𝄢 Mozilla https://www.mozilla.org/ 𝄂 Before I built a wall I'd ask to know What I was walling in or walling out, And to whom I was like to give offense. - Robert Frost, Mending Wall (1914)
signature.asc
Description: Digital signature
_______________________________________________ dev-platform mailing list dev-platform@lists.mozilla.org https://lists.mozilla.org/listinfo/dev-platform
