On Fri, Dec 12, 2014 at 09:32:09AM -0800, jmaher wrote: > In the history of running Talos, there has never been an easy way to > determine if your change has fixed a regression or created a new one. We > have compare.py and compare-talos which are actually quite useful, but it > requires you to run yet another tool - in short you have to break your normal > workflow. > > Over the last year of looking at Talos regressions, it usually is pretty > obvious within 3 data points if you have a real sustained regression. There > are many reasons for needing >1 data point, no need to go into that here. > When you land a change to an integration tree, the automated alerts will wait > until 12 future data points are available and then determine if there is a > sustained regression or improvement. Can we do this in a more streamlined > way? > > I believe so, the easy way is to abuse the 'orange' color for a job on > treeherder. The way I see it working is like this: > * push to try, include some talos jobs in your patch > * at the end of your job after data is uploaded, we query graph server for > the try data and compare it to the expected range of data in mozilla-central > for the last 7 days. > * If your data point is outside of the range we turn the job orange with a > message to retrigger this job a couple more times > * When there are at least 3 data points (read, most likely 3 orange talos > jobs on your try push) the message will indicate you probably have a > sustained regression. > > This pattern makes a few assumptions: > 1) That you will run Talos on try server > 2) That you are fine with orange Talos jobs and manually retriggering > 3) That we turn improvements into oranges as well
There's a fourh assumption: 4) That your push is based on a recent mozilla-central tree. Most of the time, it's a valid assumption, but it's still worth noting. > I would like to know if this as a hack would be useful to many. Yes, totally. Mike _______________________________________________ dev-platform mailing list dev-platform@lists.mozilla.org https://lists.mozilla.org/listinfo/dev-platform
