On 2014-12-12 12:32 PM, jmaher wrote:
In the history of running Talos, there has never been an easy way to determine
if your change has fixed a regression or created a new one. We have compare.py
and compare-talos which are actually quite useful, but it requires you to run
yet another tool - in short you have to break your normal workflow.
Over the last year of looking at Talos regressions, it usually is pretty obvious
within 3 data points if you have a real sustained regression. There are many
reasons for needing >1 data point, no need to go into that here. When you land
a change to an integration tree, the automated alerts will wait until 12 future
data points are available and then determine if there is a sustained regression or
improvement. Can we do this in a more streamlined way?
I believe so, the easy way is to abuse the 'orange' color for a job on
treeherder. The way I see it working is like this:
* push to try, include some talos jobs in your patch
* at the end of your job after data is uploaded, we query graph server for the
try data and compare it to the expected range of data in mozilla-central for
the last 7 days.
* If your data point is outside of the range we turn the job orange with a
message to retrigger this job a couple more times
* When there are at least 3 data points (read, most likely 3 orange talos jobs
on your try push) the message will indicate you probably have a sustained
regression.
This pattern makes a few assumptions:
1) That you will run Talos on try server
2) That you are fine with orange Talos jobs and manually retriggering
3) That we turn improvements into oranges as well
I would like to know if this as a hack would be useful to many. Quite possibly
there are other ways to solve this problem in a way that isn't so hacky, please
let us know. I believe in the longer term (2-6 months) we could have a view on
treeherder that does a lot of this for us.
I think this is a great plan!
_______________________________________________
dev-platform mailing list
dev-platform@lists.mozilla.org
https://lists.mozilla.org/listinfo/dev-platform
