On 20/08/14 17:37, Jonas Sicking wrote:
It would however be really cool if we were able to pull data on which tests tend to fail in a way that affects all platforms, and which ones tend to fail on one platform only.
Here's a potential project that might help. For all of the trees (probably try especially), look at the checkins and for each directory affected build up a probability of failure for each of the tests.
You would have to find which commits were on m-c at the time of the run to set the baseline for the checkin; and intermittent failures would add a certain noise floor.
The basic idea though is that the information would be very simple to use: For each directory touched in a commit, find all the tests that cross a certain failure threshold across the assembled dataset and ensure that those test groups are run.
And this would need to include prerequisites, like builds for the given runs. You would, of course, include builds as tests.
Setting the threshold might take some tuning, because failure rates will vary across different test groups. I keep hearing bad things about certain ones, for instance and build failures are far less common than test failures on the whole, naturally.
_______________________________________________ dev-platform mailing list dev-platform@lists.mozilla.org https://lists.mozilla.org/listinfo/dev-platform
