On 2013-06-25 1:42 PM, Clint Talbert wrote:
On 6/24/2013 8:02 PM, Justin Lebar wrote:
Under what circumstances would you expect the code coverage build to
break but all our other builds to remain green?
Sorry, I should have been more clear. For builds, I think it would be
pretty unusual for them to break on code coverage and yet remain green
on non-coverage builds. But I've seen tests do wild things with code
coverage enabled because timing changes so much. The most worrisome
issues are when tests cause crashes on coverage enabled builds, and
that's what I'm looking for help in tracking down and fixing. Oranges on
coverage enabled builds I can live with (because they don't change
coverage numbers in large ways and can even point us at timing dependent
tests, which could be a good thing in the long run), but crashes
effectively prevent us from measuring coverage for that test
suite/chunk. Test crashes were one of the big issues with the old system
-- we could never get eyes on the crashes to debug what had happened and
get it fixed.
If such crashes are due to changes in the timings of things happening
inside the tests, then they're probably crashes that our users might
also experience, and I would expect them to be treated with the same
attention as other such crashes.
Cheers,
Ehsan
_______________________________________________
dev-platform mailing list
dev-platform@lists.mozilla.org
https://lists.mozilla.org/listinfo/dev-platform
