Sorry there's the best option left 4. make things faster
^ this is super helpful if achievable. We have also done plenty of work on this like pre-compiling spark. I believe there's still work to do but I don't believe 4. itself can solve our current issue. On Wed, Oct 28, 2026 at 2:19 PM Tian Gao <[email protected]> wrote: > There are only 3 ways to deal with it: > 1. less coverage > 2. move the workload > 3. increase budget > > We can carefully do 1. and we have done plenty of 1., but it's slow > (otherwise risky). We did pre-merge for 2. and like Dongjoon said, moving > all the workload to committers will cost committers' budget - it's free but > the number of concurrent jobs is limited. Also I'm a bit worried that if we > "outsource" all our checks, we don't have a convincing way to make sure out > code is really working. > > We are exploring the possibility of 3. at this point. > > Tian > > On Wed, Oct 28, 2026 at 2:08 PM Nicholas Chammas < > [email protected]> wrote: > >> >> On Oct 28, 2026, at 6:36 PM, Hyukjin Kwon <[email protected]> wrote: >> >> We're facing resource issues in CI. More than 50% is from the post builds. >> >> >> Just to be clear, do you mean the builds of commits after they’ve landed >> on master? >> >> Also, do you have a rough analysis available anywhere to review? Someone >> from ASF Infra shared an analysis that seemed to indicate the biggest >> source of GitHub Actions usage was from our scheduled builds, not master. >> >> I have an idea to leverage committers' fork when they merge, triggering a >> job in committers fork repo. The capacity might be an issue. WDYT about it? >> >> >> I haven’t thought this through fully, but I feel like we should explore >> the idea of a custom merge queue >> <https://docs.github.com/en/repositories/configuring-branches-and-merges-in-your-repository/configuring-pull-request-merges/managing-a-merge-queue>. >> We mark a PR as ready to merge, and an automated process tests it against >> the latest master (using the fork’s GHA budget) and merges it in >> automatically on success. If a PR fails the build, it gets removed from the >> queue. >> >> We already build PRs in the fork context; we just need to automate the >> process of testing against the latest master + merging it in. Backports + >> Jira management can continue to be driven manually by the merge script. >> >> Nick >> >>
