On 12-08-18 9:42 AM, ja...@hoppipolla.co.uk wrote:
On Friday, 17 August 2012 23:38:22 UTC+2, Justin Dolske wrote:
I'm talking about the problem of having a large set of tests with a
small percentage that fail intermittently, which is what we have today
in m-c. Even if they all magically became cross-browser compatible right
now, I think it could still be a tough sell to get other browsers
vendors to run them. The rarity of a successful all-green run means you
need people (like our tree sheriffs) to interpret what is a known
failures and what is a real problem. AIUI Chromium has similar issues
(any know about MS/Apple/Opera?), so if we were importing their tests
we'd have to decide if that was worth it.
I know about Opera ;)
We have experienced the same kind of problems with randomly failing tests that you have,
both in tests we have written ourselves and tests that have been imported from other
places. We have put quite a lot of effort into fixing the problem and now have quite
extensive systems for identifying unstable tests as soon as they are added to our test
repository, and before they have the chance to cause the equivalent of "intermittent
orange". We also have ways to flag bogus changes in test status, and so can identify
frequently misbehaving tests.
As a consequence of this we have become better at writing stable tests, and by
the time we release tests we are generally pretty confident that they are
stable at least in Opera on our systems. We are also unafraid of using
externally written tests because we can detect many quality issues before they
have a chance to cost developers lots of time investigating bad results.
Any program which relies on an event loop is by definition going to
suffer from intermittent changes in behavior because of event ordering,
etc. This means that some tests will fail intermittently in any web
browser even if the browser itself is bug free, which means that any
browser engine with a substantial amount of tests needs to come up with
ways of dealing with those. Therefore, I agree with James that
intermittently failing tests should not stop us from proceeding on
sharing tests between different engines.
Cheers,
Ehsan
_______________________________________________
dev-platform mailing list
dev-platform@lists.mozilla.org
https://lists.mozilla.org/listinfo/dev-platform
