@jamesfredley who do we need permission from to use the AWS subaccount to host self hosted agents?
I think it's time we switch - it would help us and others. On 2026-10-16 06:20 UTC Robert Thomson wrote: > Hi, > > Please see this post in [email protected] mailing list regarding GitHub > Actions usage. > If this usage is something that is required, please consider looking for > donations that can be used to run self-hosted runners. > > Thanks. > Kind regards, > -Bob Thomson. > > > ---------- Forwarded message --------- > From: Kevin Liu <[email protected]> > Date: Thu, Oct 15, 2026 at 6:07 PM > Subject: GitHub Actions: one user's PR burst on grails-core is saturating > ubuntu-24.04 for the whole org > To: <[email protected]> > > > Hi Infra team, > > Linux runners in the apache org have been saturated for the past two days, > and PR CI across many projects is waiting hours for a runner. > > What's happening right now: A single user opened a batch of about 17 PRs on > a single project, apache/grails-core, around 13:07 UTC today. Each PR's CI run > is using 25–47 jobs at once. Together they hold 569 of the 576 > ubuntu-24.04 runners > in use across the org. That's above the 20-job limit in the GitHub Actions > policy <https://infra.apache.org/github-actions-policy.html>. > > Everyone else on ubuntu-24.04 is stuck behind it, including iceberg, > zeppelin, datafusion-comet, ozone and iceberg-rust. Snapshot at 16:56 UTC > across the 345 apache repos pushed to in the last 24 hours: > Label Running Queued Median wait of queued jobs > ubuntu-latest 86 1,617 41 min > ubuntu-24.04 576 1,095 129 min > ubuntu-26.04 105 439 67 min > > Why we care: iceberg-rust's merge queue was timing out on runner waits ( > apache/iceberg-rust#3442 > <https://github.com/apache/iceberg-rust/issues/3442>). Yesterday we moved > off ubuntu-latest to ubuntu-24.04, which cut our median CI time from 232 > minutes to 9 minutes. Since about 15:00 UTC today we're back to queuing for > over an hour. > > Asks: > > 1. > > Can you reach out to grails-core about capping matrix concurrency > (strategy.max-parallel: > 15 or less)? > 2. > > Is the 20-job limit enforced or monitored anywhere? A burst like this > from one user shouldn't be able to take over the shared pool. > 3. > > Separately, ubuntu-latest has had a backlog of 1,600–2,000 jobs for two > days, even while other labels were idle. Is there something specific to > that label you can see on your side? > > Happy to share the scripts and raw data. > > Best, > > Kevin >
