On 18:25, Tue, 19 Aug, Ehsan Akhgari wrote:
On 2014-08-19, 5:49 PM, Jonathan Griffin wrote:On 8/19/2014 2:41 PM, Ehsan Akhgari wrote:On 2014-08-19, 3:57 PM, Jeff Gilbert wrote:I would actually say that debug tests are more important for continuous integration than opt tests. At least in code I deal with, we have a ton of asserts to guarantee behavior, and we really want test coverage with these via CI. If a test passes on debug, it should almost certainly pass on opt, just faster. The opposite is not true."They take a long time and then break" is part of what I believe caused us to not bother with debug testing on much of Android and B2G, which we still haven't completely fixed. It should be unacceptable to ship without CI on debug tests, but here we are anyways. (This is finally nearly fixed, though there is still some work to do) I'm not saying running debug tests less often is on the same scale of bad, but I would like to express my concerns about heading in that direction.I second this. I'm curious to know why you picked debug tests for this experiment. Would it not make more sense to run opt tests on desktop platforms on every other run?Just based on the fact that they take longer and thus running them less frequently would have a larger impact. If there's a broad consensus that debug runs are more valuable, we could switch to running opt tests less frequently instead.Yep, the debug tests indeed take more time, mostly because they run more checks. :-) The checks in opt builds are not exactly a subset of the ones in debug builds, but they are close. Based on that, I think running opt tests on every other push is a more conservative one, and I support it more. That being said, for this one week limited trial, given that the sheriffs will help backfill the skipped tests, I don't care very strongly about this, as long as it doesn't set the precedence that we can ignore debug tests!
I'd like to highlight that we're still planning on running debug linux64 tests for every build. This is based on the assumption that debug-specific failures are generally cross-platform failures as well.
Does this help alleviate some concern? Or is that assumption just plain wrong?
Cheers, Chris
signature.asc
Description: Digital signature
_______________________________________________ dev-platform mailing list dev-platform@lists.mozilla.org https://lists.mozilla.org/listinfo/dev-platform
