Great question Brian! I believe you are asking if we would generate alerts based on individual tests instead of the summary of tests. The short answer is no. In looking at reporting the subtest results as new alerts, we found there was a lot of noise (especially in svgx, tp5o, dromaeo, v8) as compared to the summary alerts. cart/tart has a bit of noise which might be more realistic, to report specific pages on, we could investigate this more if there is a strong need for it.
late last year we cleaned up our summary reporting to be a geometric mean of all the pages/subtests which were run. This means that we do a better job of reporting a regression of the test summary when a specific test is the cause. One thing we have been working on is a compare mode to perfherder which replaces compare-talos. This is live ( https://treeherder.mozilla.org/perf.html#/comparechooser - although changing rapidly) and will do a great job of telling you which specific test caused a regression. Do you have concerns about a lack of reporting subtests, or tooling to make finding the results easier? suggestions are welcome, this is something we work on regularly and improve to make our lives easier! -Joel On Mon, May 4, 2015 at 2:08 PM, Brian Grinstead <bgrinst...@mozilla.com> wrote: > The upcoming changes sound great! Is there currently a way (or plans to > add a way) to track regressions / improvements for a single measurement > within a test? I see that in perfherder I can add these measurements to a > graph (http://mzl.la/1E17Zyo) but it’s hard to distinguish between normal > variation across runs and an actual regression by looking at the graph. > > Brian > > _______________________________________________ dev-platform mailing list dev-platform@lists.mozilla.org https://lists.mozilla.org/listinfo/dev-platform
