andygrove opened a new pull request, #5843: URL: https://github.com/apache/datafusion-comet/pull/5843
## Which issue does this PR close? Closes #5838. ## Rationale for this change CI load on pull requests is dominated by suites that don't really need to be green until merge time. A merge queue lets us move them there: the queue tests the exact tree that will land, so an iterating PR can run a smaller set without losing any coverage on `main`. The part I expected to be a blocker turned out not to be. Merge queue is self-serve in `.asf.yaml` now — rulesets there accept a raw GitHub Rulesets API payload and `asfyaml/feature/github/rulesets.py` passes it through with no rule-type allowlist, so a `merge_queue` rule needs no INFRA ticket. apache/iceberg-python, apache/seatunnel and apache/texera all run queues configured this way. Worth noting apache/datafusion is _not_ the template even though it's the obvious place to look: their `.asf.yaml` has no queue config and `gh api repos/apache/datafusion/rulesets` returns `[]`, because INFRA enabled theirs directly in classic branch protection via INFRA-27154 before asfyaml supported rulesets. ## What changes are included in this PR? CI splits into two tiers: | | PR tier | queue tier | push to main | | --- | --- | --- | --- | | build Linux/macOS, benchmark, Spark 4.1, Iceberg 1.11 | ✓ | ✓ | — | | Spark 3.4 / 3.5 / 4.0, Iceberg 1.8 / 1.9 / 1.10 | label only | ✓ | — | | docs deploy | — | — | ✓ | Spark 3.5 is the one that changes for contributors: it ran on every PR and is now queue-only, with a `run-spark-3.5-tests` label as an escape hatch. Spark 4.1 stays in the PR tier because it's the default build profile, so it's the cheapest early warning that something is wrong. Dropping the `push` arm from the heavy jobs is load-bearing rather than cleanup. The queue already tested the tree that lands, so keeping the old push-to-main runs would make every merge pay for CI twice and total usage would go *up*, not down. Iceberg 1.8/1.9/1.10 move into the queue for the same reason — they were push-only, so otherwise they'd have nowhere left to run. On the `.asf.yaml` side: a `Merge Queue` ruleset targeting `~DEFAULT_BRANCH` (merge queue rules reject wildcard ref patterns, so release branches keep plain protection), `Required Checks` as the single required status check, and `allow_auto_merge` so a PR can be queued before its checks finish. `apache/root` is a bypass actor so a wedged queue can be recovered without a ticket. GitHub evaluates the ruleset and classic branch protection together, so the existing one-approving-review requirement is unaffected. The queue parameters worth arguing about are `max_entries_to_merge: 5` with `max_entries_to_build: 2` and no minimum wait — no added latency when the queue is empty, but once it backs up a single pipeline validates up to five PRs. Easy to tune later. ## How are these changes tested? `preflight` runs `dev/ci/check-ci-config.py`, which (as of the previous commit) enforces that every job is registered with the aggregator and that the `Required Checks` name matches the context this PR adds to `.asf.yaml`. `actionlint`, `prettier --check "**/*.md"` and a YAML parse of both changed configs are clean. The `merge_group` paths can't really be exercised until the ruleset is live, which is the main reason this is split out from #5842 — that PR lands `Required Checks` on its own so we can confirm it reports correctly on real PRs first. ## Notes for reviewers - Stacked on top of the `Required Checks` PR, so it shows both commits until that one merges. **Do not merge this before it.** - `run-spark-3.4-tests` and `run-spark-3.5-tests` need creating in repository settings. 3.4's label doesn't exist today either, so that escape hatch has quietly been doing nothing. - Once these suites gate the queue, a flaky test blocks everyone's merges rather than one PR. datafusion sent a dev@ note before enabling theirs and we should probably do the same. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
