In the history of running Talos, there has never been an easy way to determine if your change has fixed a regression or created a new one. We have compare.py and compare-talos which are actually quite useful, but it requires you to run yet another tool - in short you have to break your normal workflow.
Over the last year of looking at Talos regressions, it usually is pretty obvious within 3 data points if you have a real sustained regression. There are many reasons for needing >1 data point, no need to go into that here. When you land a change to an integration tree, the automated alerts will wait until 12 future data points are available and then determine if there is a sustained regression or improvement. Can we do this in a more streamlined way? I believe so, the easy way is to abuse the 'orange' color for a job on treeherder. The way I see it working is like this: * push to try, include some talos jobs in your patch * at the end of your job after data is uploaded, we query graph server for the try data and compare it to the expected range of data in mozilla-central for the last 7 days. * If your data point is outside of the range we turn the job orange with a message to retrigger this job a couple more times * When there are at least 3 data points (read, most likely 3 orange talos jobs on your try push) the message will indicate you probably have a sustained regression. This pattern makes a few assumptions: 1) That you will run Talos on try server 2) That you are fine with orange Talos jobs and manually retriggering 3) That we turn improvements into oranges as well I would like to know if this as a hack would be useful to many. Quite possibly there are other ways to solve this problem in a way that isn't so hacky, please let us know. I believe in the longer term (2-6 months) we could have a view on treeherder that does a lot of this for us. Thanks for reading- lets make things more useful! _______________________________________________ dev-platform mailing list dev-platform@lists.mozilla.org https://lists.mozilla.org/listinfo/dev-platform
