Thanks James, nice work.
On Thu, Oct 22, 2026 at 6:51 PM James Daugherty <[email protected]> wrote: > > Hi Bob, > > We've made some changes that should reduce our usage of CI by half or more. > We're still discussing the next steps to reduce further. > > Regards, > James > > On 2026-10-16 06:20 UTC Robert Thomson wrote: > > Hi, > > > > Please see this post in [email protected] mailing list regarding GitHub > > Actions usage. > > If this usage is something that is required, please consider looking for > > donations that can be used to run self-hosted runners. > > > > Thanks. > > Kind regards, > > -Bob Thomson. > > > > > > ---------- Forwarded message --------- > > From: Kevin Liu <[email protected]> > > Date: Thu, Oct 15, 2026 at 6:07 PM > > Subject: GitHub Actions: one user's PR burst on grails-core is saturating > > ubuntu-24.04 for the whole org > > To: <[email protected]> > > > > > > Hi Infra team, > > > > Linux runners in the apache org have been saturated for the past two days, > > and PR CI across many projects is waiting hours for a runner. > > > > What's happening right now: A single user opened a batch of about 17 PRs on > > a single project, apache/grails-core, around 13:07 UTC today. Each PR's CI > > run > > is using 25–47 jobs at once. Together they hold 569 of the 576 > > ubuntu-24.04 runners > > in use across the org. That's above the 20-job limit in the GitHub Actions > > policy <https://infra.apache.org/github-actions-policy.html>. > > > > Everyone else on ubuntu-24.04 is stuck behind it, including iceberg, > > zeppelin, datafusion-comet, ozone and iceberg-rust. Snapshot at 16:56 UTC > > across the 345 apache repos pushed to in the last 24 hours: > > Label Running Queued Median wait of queued jobs > > ubuntu-latest 86 1,617 41 min > > ubuntu-24.04 576 1,095 129 min > > ubuntu-26.04 105 439 67 min > > > > Why we care: iceberg-rust's merge queue was timing out on runner waits ( > > apache/iceberg-rust#3442 > > <https://github.com/apache/iceberg-rust/issues/3442>). Yesterday we moved > > off ubuntu-latest to ubuntu-24.04, which cut our median CI time from 232 > > minutes to 9 minutes. Since about 15:00 UTC today we're back to queuing for > > over an hour. > > > > Asks: > > > > 1. > > > > Can you reach out to grails-core about capping matrix concurrency > > (strategy.max-parallel: > > 15 or less)? > > 2. > > > > Is the 20-job limit enforced or monitored anywhere? A burst like this > > from one user shouldn't be able to take over the shared pool. > > 3. > > > > Separately, ubuntu-latest has had a backlog of 1,600–2,000 jobs for two > > days, even while other labels were idle. Is there something specific to > > that label you can see on your side? > > > > Happy to share the scripts and raw data. > > > > Best, > > > > Kevin > >
