On Sat, Mar 1, 2025 at 2:36 PM Paolo Bonzini <pbonz...@redhat.com> wrote: > > On 3/1/25 07:19, Stefan Hajnoczi wrote: > > Hi, > > On February 26th GitLab CI started failing many jobs because they > > could not be scheduled. I've been unable to merge pull requests > > because the CI is not working. > > > > Here is an example failed job: > > https://gitlab.com/qemu-project/qemu/-/jobs/9281757413 > > Hi Stefan, > > until February 26th the Digital Ocean runners were not enabled; I tried > enabling them (which is what caused the issue) to start gauging how much > credit we would need to be able to move from Azure to DO for CI. I > posted a note on IRC, I'm sorry if you missed that. > > > The cache PVC appears to be a manual addition made to the running > > cluster but not committed to qemu.git. I don't understand why the > > problems only started surfacing now. Maybe a recent .gitlab-ci.d/ > > change changed how the timeout behaves or maybe the gitlab-runner > > configuration that enables the cache PVC simply wasn't picked up by > > the gitlab-runner Pod until February 26th? > > Almost: the cache is not used on Azure, which is why it works. > > > In the short term I made a manual edit to the ConfigMap removing > > gitlab-cache-pvc (but I didn't delete the PVC resource itself). Jobs > > are at least running now, although they may take longer due to the > > lack of cache. > > Ok, thanks for debugging that. I think what you did is right, and the > caching setup should be tested more on a secondary cluster.
Glad the change is acceptable and didn't break things more. > (As to the DO credits numbers, the cost of the k8s cluster is about > $75/month, and since we were granted $2000 in credits we have only > $1100/year to spend on the actual jobs. The plan is to check on the > credits left at the end of March and bring our estimates to DO's open > source program manager). This reminds me I received an email asking for feedback regarding QEMU's Amazon credits. Just wanted to mention they are there if we need them. Stefan
