On 01/03/2019 09:51 AM, Jonathan Kew wrote:
On 03/01/2019 16:17, jmaher wrote:
What are the risks associated with this?
1) try server build times will increase as we will be testing on PGO
instead of OPT
2) we could miss a regression that only shows up on OPT, but if we
only ship PGO and once we leave central we do not build OPT, this is
a very low risk.
It's not just tryserver build times. Presumably this will also tend to
increase the time between a patch landing on inbound or autoland and
any resulting test failures showing up.
This seems like a negative in that it means more patches are likely to
have landed on top of the regressing one in the meantime, potentially
complicating backouts, and the original developer may be less likely
to still be around for a quick investigation/fix.
How long does it typically take for full PGO test results to be
available for a push? How does that compare to full Opt test results?
ISTM that if the increase is quite marginal, this may be OK, but if
the latency becomes substantially greater, there will be a continual
cost in increased developer and/or sheriff pain.
Good points, but given that most failures will show up debug builds, it
seems like a more relevant metric is the difference between time(Opt) vs
min(time(debug), time(PGO)). Though debug builds may run slow enough
that it boils down to what you said?
Looking at Windows 64-bit jobs from a random push (
https://treeherder.mozilla.org/#/jobs?repo=mozilla-inbound&revision=63027ff03effb04ed4bf53bbb0c9aa1bad4b4c9b
), I see:
pgo: build=119min + Wd1=15min
opt: build=55min + Wd1=13min
debug: build=46min + Wd1=22min
So by that, you get opt and debug Wd1 results back at the same time
(67-68min) and pgo Wd1 results take twice as long (134min). I imagine
there are much slower test jobs that make this situation cloudier, but
assuming the general pictures holds then it seems like opt is mostly
redundant with debug.
The majority of your currently opt-triggered backouts will still happen,
just using debug results now. This is assuming debug normally catches a
superset of the problems that opt would, which is asserted in #1 of
jmaher's post.
+1 from me for killing off opt tests.
_______________________________________________
dev-platform mailing list
dev-platform@lists.mozilla.org
https://lists.mozilla.org/listinfo/dev-platform
