On Friday, 17 August 2012 23:38:22 UTC+2, Justin Dolske wrote: > I'm talking about the problem of having a large set of tests with a > small percentage that fail intermittently, which is what we have today > in m-c. Even if they all magically became cross-browser compatible right > now, I think it could still be a tough sell to get other browsers > vendors to run them. The rarity of a successful all-green run means you > need people (like our tree sheriffs) to interpret what is a known > failures and what is a real problem. AIUI Chromium has similar issues > (any know about MS/Apple/Opera?), so if we were importing their tests > we'd have to decide if that was worth it.
I know about Opera ;) We have experienced the same kind of problems with randomly failing tests that you have, both in tests we have written ourselves and tests that have been imported from other places. We have put quite a lot of effort into fixing the problem and now have quite extensive systems for identifying unstable tests as soon as they are added to our test repository, and before they have the chance to cause the equivalent of "intermittent orange". We also have ways to flag bogus changes in test status, and so can identify frequently misbehaving tests. As a consequence of this we have become better at writing stable tests, and by the time we release tests we are generally pretty confident that they are stable at least in Opera on our systems. We are also unafraid of using externally written tests because we can detect many quality issues before they have a chance to cost developers lots of time investigating bad results. > Given the long history (shall I say "plague"?) of intermittent-orange in > our tree, I can't agree that this would be a non-issue or is easy to > fix! [Nor am I saying reusable tests are a bad idea -- just that it > would seem wise to ramp up over time.] >From our point of view it is no problem to import tests even if they are not >100% stable (and of course if the instability is due to a bug in Opera they >could be stable for you and not for us). The worst case scenario is that we >will simply disable the problematic tests but continue to get the benefit of >all the other tests. That is a much better position to be in than not being >able to run the tests at all. _______________________________________________ dev-platform mailing list dev-platform@lists.mozilla.org https://lists.mozilla.org/listinfo/dev-platform
