On 2014-04-09, 6:46 PM, Chris Peterson wrote:
On 4/9/14, 11:48 AM, Gregory Szorc wrote:
I feel a lot of people just shrug shoulders and allow the test to be
disabled (I'm guilty of it as much as anyone). From my perspective, it's
difficult to convince the powers at be that fixing intermittent failures
(that have been successfully swept under a rug and are out of sight and
out of mind) is more important than implementing some shiny new feature
(that shows up on an official goals list). I feel we all need to treat
failing tests with more urgency. The engineering culture is not
currently favoring that.
How can we realign our testing priorities, short of some management
decree?
Speaking from my experience on this over the years, I don't think we can
(and we're working on that!)
Sheriffs have had to disable entire test suites, but even that
has not motivated test owners to fix their intermittent tests. Part of
the problem, I understand, is that many tests have no owner, so the
responsibility for even debugging test failures is not clear.
How much effort do we want to spend creating tools to manage and rerun
known intermittent tests compared to diagnosing the root problems?
Earlier in this thread, Ehsan said that, once upon a time, we had no
intermittent tests. Someone else suggested (half-jokingly, I assume :)
that requiring a 100% green TBPL before landing on inbound would be an
effective motivator to fix intermittent tests.
I am not convinced we need many additional tools at this point. All of
the tools in the world won't help unless we have people using them!
Without that, I think discussing what tools to build might be premature.
Cheers,
Ehsan
_______________________________________________
dev-platform mailing list
dev-platform@lists.mozilla.org
https://lists.mozilla.org/listinfo/dev-platform
