On Tue, Aug 19, 2014 at 02:49:48PM -0700, Jonathan Griffin wrote: > On 8/19/2014 2:41 PM, Ehsan Akhgari wrote: > >On 2014-08-19, 3:57 PM, Jeff Gilbert wrote: > >>I would actually say that debug tests are more important for continuous > >>integration than opt tests. At least in code I deal with, we have a ton > >>of asserts to guarantee behavior, and we really want test coverage with > >>these via CI. If a test passes on debug, it should almost certainly pass > >>on opt, just faster. The opposite is not true. > >> > >>"They take a long time and then break" is part of what I believe caused > >>us to not bother with debug testing on much of Android and B2G, which we > >>still haven't completely fixed. It should be unacceptable to ship > >>without CI on debug tests, but here we are anyways. (This is finally > >>nearly fixed, though there is still some work to do) > >> > >>I'm not saying running debug tests less often is on the same scale of > >>bad, but I would like to express my concerns about heading in that > >>direction. > > > >I second this. I'm curious to know why you picked debug tests for this > >experiment. Would it not make more sense to run opt tests on desktop > >platforms on every other run? > > > Just based on the fact that they take longer and thus running them less > frequently would have a larger impact. If there's a broad consensus that > debug runs are more valuable, we could switch to running opt tests less > frequently instead.
It seems to me our goal here is basically to pick so that the expected time to detect bustage is minimized without increasing the maximum time it can take to detect bustage. That is take p(d) to be the probability only debug tests will fail, p(o) the probability only opt tests will fail, and p(b) the probability both will fail. Then take t(d) and t(o) the time for a debug and opt test to run respectively. Now you want to decide which to run first debug or opt. you'd expect that if you choose debug you'd expect to detect bustage in (p(d) + p(b)) * t(d) + p(o) * (t(o) + t(d)) which simplifies to t(d) + p(o) * t(o) On the other hand if you choose to test opt first you get t(o) + p(d) * t(d) I suspect we all agree t(d) > t(o) and it seems likely p(d) > p(o), but it should be clear which is the better choice depends on the exact values of those numbers (and this is not a good model of reality in many ways). Trev > > Jonathan > _______________________________________________ > dev-platform mailing list > [email protected] > https://lists.mozilla.org/listinfo/dev-platform _______________________________________________ dev-platform mailing list [email protected] https://lists.mozilla.org/listinfo/dev-platform
