-1 to Hyukjin’s proposal to run post-merge builds in committers’ forks. I already struggle to get enough CI resources for my own PRs, so I do not have spare capacity for post-merge builds.
+0.5, rather than +1, to Nicholas’s suggestion that we strengthen pre-merge CI against a recent master. I agree that pre-merge CI should remain the primary gate and preserve the existing coverage for every PR. The merge script could reject a PR if its successfully tested revision is more than 30 commits behind the target branch, asking the contributor to update it and rerun CI. However, I do not think this can completely replace post-merge CI. A PR may still be tested against a state different from the one into which it is ultimately merged, especially when several PRs merge concurrently. Pre-merge CI against master also does not validate subsequent backports. Therefore, I would keep post-merge CI, but reduce its cost through narrower per-commit testing and periodic full builds. For post-merge CI on *master*, I suggest combining narrower test selection with periodic full builds: - For nine out of every ten merged commits, run a narrow set of tests selected from that commit’s changes. - For every tenth commit, run the full build_main suite. This would provide feedback for every commit while using periodic full builds to catch tests missed by selective testing. For *branch-4.x*, I suggest retaining its current one-in-ten behavior. <https://github.com/apache/spark/pull/57841> For *older release branches*, I would initially keep post-merge CI on every commit to preserve backport coverage. This is not free, however: in the latest 1,000 build_main <https://github.com/apache/spark/actions/workflows/.github/workflows/build_main.yml> runs, master consumed about 49.8% of runner time, branch-4.x 5.1%, and the older release branches together about 45.0%. We may therefore need to revisit the older branches if reducing master is insufficient. For scheduled jobs, perhaps we could start with a weekly baseline and temporarily increase a relevant job to daily during active development. For example, build_python_3.12_pandas_3 could run daily while pandas 3 support is actively changing and return to weekly afterward. Regarding test selection, would we be open to experimenting with AI-based approaches? I have a POC in #57461 <https://github.com/apache/spark/pull/57461> that uses the diff and repository context to select likely affected test suites. Our existing deterministic selection primarily operates at the module or job-group level; it cannot select individual test suites at the same granularity. Once a module is considered affected, a relatively broad set of tests may still run. My impression is that maintaining a prompt might be simpler than continuing to expand those rules. The AI selection would be approximate and nondeterministic. We could first evaluate it against full builds, measuring its selection quality and runner cost. When a full build exposes a relevant test that the AI omitted, we could use that case to refine the prompts. Periodic full runs would both catch remaining gaps and provide feedback for improving future selections. On Thu, Oct 29, 2026 at 5:03 PM Nicholas Chammas <[email protected]> wrote: > > On Oct 28, 2026, at 11:19 PM, Tian Gao via dev <[email protected]> > wrote: > > 2. move the workload > > > The nice thing about the merge queue idea is that the build is already > being run on contributors’ forks as part of PR CI. With the added > constraint that it runs against the latest master right before merge, we > can eliminate the need for post-build merges entirely. > > 3. increase budget > > > Is this a reference to ASF’s concept of targeted sponsorship? It would be > great if companies could donate specifically to support Spark’s CI budget. > ASF’s page on this seems to indicate it is possible > <https://www.apache.org/foundation/docs/targeted-sponsorship-policy.html#:~:text=Continuous%20Integration> > . > > > 4. make things faster > > Also: Make the build more selective > <https://issues.apache.org/jira/browse/SPARK-59180> (which is one way to > make it faster). > > -- Ruifeng Zheng E-mail: [email protected]
