Hi Joel,
jmaher wrote:
I believe so, the easy way is to abuse the 'orange' color for a job on
treeherder. The way I see it working is like this:
* push to try, include some talos jobs in your patch
* at the end of your job after data is uploaded, we query graph server for the
try data and compare it to the expected range of data in mozilla-central for
the last 7 days.
* If your data point is outside of the range we turn the job orange with a
message to retrigger this job a couple more times
* When there are at least 3 data points (read, most likely 3 orange talos jobs
on your try push) the message will indicate you probably have a sustained
regression.
This pattern makes a few assumptions:
1) That you will run Talos on try server
2) That you are fine with orange Talos jobs and manually retriggering
3) That we turn improvements into oranges as well
I would like to know if this as a hack would be useful to many. Quite possibly
there are other ways to solve this problem in a way that isn't so hacky, please
let us know. I believe in the longer term (2-6 months) we could have a view on
treeherder that does a lot of this for us.
This would be very useful! What I basically want to know when I push to
try is "will this cause a regression that will result in Talos
regression emails being sent out if I land it on inbound". Comparing
the results to the latest mozilla-central and turning the job orange if
outside the expected range will let me be much more confident about that.
Do you think it is worth doing a couple of re-triggers automatically on
the first push?
_______________________________________________
dev-platform mailing list
dev-platform@lists.mozilla.org
https://lists.mozilla.org/listinfo/dev-platform
