On 11/16/21 17:49, Daniel P. Berrangé wrote: > On Tue, Nov 16, 2021 at 05:33:09PM +0100, Thomas Huth wrote: >> The jobs on Cirrus-CI sometimes get delayed quite a bit, waiting to >> be scheduled, so while the build test itself finishes within 60 minutes, >> the total run time of the jobs can be longer due to this waiting time. >> Thus let's increase the timeout on the gitlab side a little bit, so >> that these jobs are not marked as failing just because of the delay. > > On a successful pipeline I see > > freebsd-11 - 28 minutes > freebsd-12 - 57 minutes > macos - 30 minutes > > We know cirrus allows 2 concurrent jobs, so from that I infer > that the freebsd-12 job was queued for ~30 minutes waiting for > either the freebsd-11 or macos job to finish, and then it > ran in 30 minutes, giving the ~60 minute total. > > That's too close to the 60 minute gitlab default job timeout > for comfort - it can easily slip over 60 minutes by just a > small amount. > > 80 minutes will certainly help in the case where we > randomly take a little longer than 30 minutes to build, > and have 1 of the 3 jobs queued. > > When we're running jobs on both master + staging, we can > have 2 jobs running and 4 more queued - 2 of those queued > might just finish in time, but 2 will definitely fail. > My patch will cut these extra jobs on master, so in common > case we only ever get 1 queued, which should work well in > combo with your patch here. That should be good enough > for the qemu-project namespace, unless someone is triggering > pipelines for stable branch staging at the same time as > the master branch staging. > > If we do want to worry about more than 2 queued jobs > again for that reason, we might consider putting > it upto 100 minutes. That would give us enough slack to > have 4 queued jobs behind two running jobs and have > them all succeed > >> Signed-off-by: Thomas Huth <th...@redhat.com> >> --- >> .gitlab-ci.d/cirrus.yml | 1 + >> 1 file changed, 1 insertion(+) >> >> diff --git a/.gitlab-ci.d/cirrus.yml b/.gitlab-ci.d/cirrus.yml >> index e7b25e7427..22d42585e4 100644 >> --- a/.gitlab-ci.d/cirrus.yml >> +++ b/.gitlab-ci.d/cirrus.yml >> @@ -14,6 +14,7 @@ >> stage: build >> image: registry.gitlab.com/libvirt/libvirt-ci/cirrus-run:master >> needs: [] >> + timeout: 80m >> allow_failure: true >> script: >> - source .gitlab-ci.d/cirrus/$NAME.vars > > Whether 80 or 100 minute, consider it > > Reviewed-by: Daniel P. Berrangé <berra...@redhat.com>
This pipeline took 1h51m09s: https://gitlab.com/qemu-project/qemu/-/pipelines/409666733/builds But Richard restarted unstable jobs, which probably added time to the total. IIRC from a maintainer perspective 1h15 is the upper limit. 80m fits, 100m is over. Up to the project maintainers (personally I don't have any objection, in particular if this reduces the failures rate). Reviewed-by: Philippe Mathieu-Daudé <phi...@redhat.com>
