On Wednesday, July 25, 2012 3:05:07 AM UTC+3, bent wrote: > However, I firmly believe that disabling a randomly-failing test is a > decision that should be made by the module owners and peers. > > Perhaps it's only a recent development but I've seen my OOP test suite > for IndexedDB disabled twice in the last week without anyone > consulting the module owner or peers. There's no real way that any of > our sheriffs or volunteers can know all the ramifications of disabling > a particular test (and it shouldn't be their responsibility to find > out - nobody has that much time!). The solution is simple though: > request review from a module peer before disabling a test. Peers can > make the decision if it's worth disabling a test, pull someone off of > other tasks to fix it, or whatever.
Peers are in charge of everything relating to their modules. As soon as tests start randomly failing, however, that places a burden on people who do *not* work on the affected module. I don't think peers should have the final say on whether their module's tests are important enough to be a nuisance to everyone else. If the people in charge of keeping trees green think random orange is too annoying, it strikes me that they're well within their rights to disable it. Of course they should certainly *inform* module peers when they do it, such as by CCing them on the relevant bug. If the test is important, then of course module peers can re-enable it -- after fixing the random orange. But no, it doesn't seem to me that peers should be able to decide that their tests are important enough that everyone else has to live with random orange because of them. I think we'd benefit from better ways of disabling tests, though. If we could mark only a particular pattern of failures as ignored, and preferably have some way to auto-log when that pattern gets hit (like to Bugzilla), we'd be able to hide random orange without reducing our test coverage so much. Currently you have to disable on a per-file basis if you don't understand how the test file works, which is bad if the file contains lots of tests or the failure is very specific. _______________________________________________ dev-platform mailing list dev-platform@lists.mozilla.org https://lists.mozilla.org/listinfo/dev-platform
