On 8/16/12 3:01 PM, Kyle Huey wrote:

The basic idea here sounds worthy, but one concern is that our own tests
are often unreliable in our own browser -- and I'd expect that to only get
worse as other browsers and their tests enter the picture.

That's not really true.  Most of our mochitests and reftests are pretty
solid.  I don't think reliability will be a huge issue, and reliability
problems due to bad tests can be fixed quite easily.

Sure, on an individual basis most tests are solid.

I'm talking about the problem of having a large set of tests with a small percentage that fail intermittently, which is what we have today in m-c. Even if they all magically became cross-browser compatible right now, I think it could still be a tough sell to get other browsers vendors to run them. The rarity of a successful all-green run means you need people (like our tree sheriffs) to interpret what is a known failures and what is a real problem. AIUI Chromium has similar issues (any know about MS/Apple/Opera?), so if we were importing their tests we'd have to decide if that was worth it.

Given the long history (shall I say "plague"?) of intermittent-orange in our tree, I can't agree that this would be a non-issue or is easy to fix! [Nor am I saying reusable tests are a bad idea -- just that it would seem wise to ramp up over time.]

Justin
_______________________________________________
dev-platform mailing list
dev-platform@lists.mozilla.org
https://lists.mozilla.org/listinfo/dev-platform

Reply via email to