On 2014-08-20, 12:02 PM, Chris AtLee wrote:
On 18:25, Tue, 19 Aug, Ehsan Akhgari wrote:
On 2014-08-19, 5:49 PM, Jonathan Griffin wrote:
On 8/19/2014 2:41 PM, Ehsan Akhgari wrote:
On 2014-08-19, 3:57 PM, Jeff Gilbert wrote:
I would actually say that debug tests are more important for
continuous integration than opt tests. At least in code I deal with,
we have a ton of asserts to guarantee behavior, and we really want
test coverage with these via CI. If a test passes on debug, it should
almost certainly pass on opt, just faster. The opposite is not true.
"They take a long time and then break" is part of what I believe
caused us to not bother with debug testing on much of Android and
B2G, which we still haven't completely fixed. It should be
unacceptable to ship without CI on debug tests, but here we are
anyways. (This is finally nearly fixed, though there is still some
work to do)
I'm not saying running debug tests less often is on the same scale of
bad, but I would like to express my concerns about heading in that
direction.
I second this. I'm curious to know why you picked debug tests for
this experiment. Would it not make more sense to run opt tests on
desktop platforms on every other run?
Just based on the fact that they take longer and thus running them less
frequently would have a larger impact. If there's a broad consensus
that debug runs are more valuable, we could switch to running opt tests
less frequently instead.
Yep, the debug tests indeed take more time, mostly because they run
more checks. :-) The checks in opt builds are not exactly a subset
of the ones in debug builds, but they are close. Based on that, I
think running opt tests on every other push is a more conservative
one, and I support it more. That being said, for this one week
limited trial, given that the sheriffs will help backfill the skipped
tests, I don't care very strongly about this, as long as it doesn't
set the precedence that we can ignore debug tests!
I'd like to highlight that we're still planning on running debug linux64
tests for every build. This is based on the assumption that
debug-specific failures are generally cross-platform failures as well.
Does this help alleviate some concern? Or is that assumption just plain
wrong?
well, yes, most of our code is cross platform, but there are debug only
checks in our platform specific code as well, so if we're talking about
something more permanent than that week long experiment, then running
debug tests on Linux64 doesn't alleviate all concerns. But it's fine
for this short experiment.
Cheers,
Ehsan
_______________________________________________
dev-platform mailing list
dev-platform@lists.mozilla.org
https://lists.mozilla.org/listinfo/dev-platform
