On 8/16/12 3:01 PM, Kyle Huey wrote:
The basic idea here sounds worthy, but one concern is that our own tests
are often unreliable in our own browser -- and I'd expect that to only get
worse as other browsers and their tests enter the picture.
That's not really true. Most of our mochitests and reftests are pretty
solid. I don't think reliability will be a huge issue, and reliability
problems due to bad tests can be fixed quite easily.
Sure, on an individual basis most tests are solid.
I'm talking about the problem of having a large set of tests with a
small percentage that fail intermittently, which is what we have today
in m-c. Even if they all magically became cross-browser compatible right
now, I think it could still be a tough sell to get other browsers
vendors to run them. The rarity of a successful all-green run means you
need people (like our tree sheriffs) to interpret what is a known
failures and what is a real problem. AIUI Chromium has similar issues
(any know about MS/Apple/Opera?), so if we were importing their tests
we'd have to decide if that was worth it.
Given the long history (shall I say "plague"?) of intermittent-orange in
our tree, I can't agree that this would be a non-issue or is easy to
fix! [Nor am I saying reusable tests are a bad idea -- just that it
would seem wise to ramp up over time.]
Justin
_______________________________________________
dev-platform mailing list
dev-platform@lists.mozilla.org
https://lists.mozilla.org/listinfo/dev-platform
