Is there an easy way for me to track what tests have been disabled as result of intermittent issues we haven't been able to fix?
On 6 September 2017 at 14:10, <jma...@mozilla.com> wrote: > Over the last 9 months a few of us have really watched intermittent test > failures almost daily and done a lot to pester people as well as fix many. > While there are over 420 bugs that have been fixed since the beginning of the > year, there are half that many (211+) which have been disabled in some form > (including turning off the jobs). > > We don't like to disable and have been pretty relaxed in recommending > disabling a test. Overall we have tried to adhere to a policy of: > * >=30 failures/week- ask for owner to look at failure and fix it, if this > persists for a few weeks with no real traction we would go ahead [and > recommend] disabling it. > * >= 75 failures/week- ask for people to fix this in a shorter time frame and > recommend disabling the test in a week or so > * >= 150 failures/week- often just disable the test > > This is confusing and hard to manage. Since then we have started adjusting > triage queries and some teams are doing their own triage and we are ignoring > those bugs (while they are getting prioritized properly). > > What we are looking to start doing this month is adopting a simpler policy: > * any bug that has >=200 instances in the last 30 days will be disabled > ** this will be a manual process, so it will happen a couple times/week > > We expect the outcome of this to be a similar amount of disabling, just an > easier method for doing so. It is very possible we might recommend disabling > a test before it hits the threshold- keep in mind a disabled test is easy to > re-enable (so feel free to disable for that one platform until you have time > to look at fixing it) > > To be clear we (and some component owners) will continue triaging bugs and > trying to get fixes in place as often as possible and prefer a fix, not a > disabled test! > > Please raise any concerns, otherwise we will move forward with this in the > coming weeks. > _______________________________________________ > dev-platform mailing list > dev-platform@lists.mozilla.org > https://lists.mozilla.org/listinfo/dev-platform _______________________________________________ dev-platform mailing list dev-platform@lists.mozilla.org https://lists.mozilla.org/listinfo/dev-platform
