hlfan left a comment (openstreetmap/openstreetmap-website#6151)
Let's take a look at the latest test run that didn't fail on "real bugs", [this
test
run](https://github.com/openstreetmap/openstreetmap-website/actions/runs/16122953585)
for #6170:
The three executions of `rails test:all` finished in 297.341805s, 281.573575s,
and 298.088535s.
Once this step finishes, the remainder of the workflow usually takes less than
15 seconds.
Now, one Ruby run fails:
- Either the other runs can complete, costing about 20 seconds of compute time
plus the re-run of one job.
- Or this run cancels the other runs, costing the re-run of two jobs.
Unless you can make the jobs take less than 20 seconds, the current variant
takes more time in this example.
---
Due to the usually parallel execution of the test runs, unless the error
happens within the first two minutes of the workflow during the setup, `rails
test:all` will only report an error after about seven minutes into the run,
when the other runs are also almost complete.
Thus, the "wasted [compute] time running tests that need to be rerun" will be
bigger **with** `fail-fast`, **unless** test-caused (not bug-caused) failures
occur more often than once in every 21 runs — based on the fact that cancelling
other jobs saves only about 20 seconds per job, while letting them finish
typically costs around 7 minutes total.
And currently, I don't see test failures being even close to the necessary
95.2% bug-caused failure rate.
Therefore, `fail-fast` currently **increases** total compute time, because most
jobs are cancelled _just seconds before completion_, and have to be re-run
anyway. So we end up paying twice.
--
Reply to this email directly or view it on GitHub:
https://github.com/openstreetmap/openstreetmap-website/pull/6151#issuecomment-3065898078
You are receiving this because you are subscribed to this thread.
Message ID:
<openstreetmap/openstreetmap-website/pull/6151/c3065898...@github.com>
_______________________________________________
rails-dev mailing list
rails-dev@openstreetmap.org
https://lists.openstreetmap.org/listinfo/rails-dev
