On 07/04/14 04:33, Andrew Halberstadt wrote:
On 06/04/14 08:59 AM, Aryeh Gregor wrote:
On Sat, Apr 5, 2014 at 12:00 AM, Ehsan Akhgari
<ehsan.akhg...@gmail.com> wrote:
Note that is only accurate to a certain point. There are other
things which
we can do to guesswork our way out of the situation for Autoland, but of
course they're resource/time intensive (basically running orange
tests over
and over again, etc.)
Is there any reason in principle that we couldn't have the test runner
automatically rerun tests with known intermittent failures a few
times, and let the test pass if it passes a few times in a row after
the first fail? This would be a much nicer option than disabling the
test entirely, and would still mean the test is mostly effective,
particularly if only specific failure messages are allowed to be
auto-retried.
Many of our test runners have that ability. But doing this implies that
intermittents are always the fault of the test. We'd be missing whole
classes of regressions (notably race conditions).
In practice how effective are we at identifying bugs that lead to
instability? Is it more common that we end up disabling the test, or
marking it as "known intermittent" and learning to live with the
instability, both of which options reduce the test coverage, or is it
more common that we realise that there is a code reason for the
intermittent, and get it fixed?
If it is the latter then making the instability as obvious as possible
makes sense, and the current setup where we run each test once can be
regarded as a compromise between the ideal setup where we run each test
multiple times and flag it as a fail if it ever fails, and the needs of
performance.
If the former is true, it makes a lot more sense to do reruns of the
tests that fail in order to keep them active at all, and store
information about the fact that reruns occurred so that we can see when
a test started giving unexpected results. This does rely on having some
mechanism to make people care about genuine intermittents that they
caused, but maybe the right way to do that is to have some batch tool
that takes all the tests that have become intermittent, and does reruns
until it has identified the commits that introduced the intermittency,
and then files P1 bugs on the developer(s) it identifies to fix their bugs.
_______________________________________________
dev-platform mailing list
dev-platform@lists.mozilla.org
https://lists.mozilla.org/listinfo/dev-platform
