andygrove opened a new pull request, #7516:
URL: https://github.com/apache/datafusion-comet/pull/7516
## Which issue does this PR close?
No issue filed. Follows on from #7311, which moved the PR tier of fork pull
requests onto the contributor's fork.
## Rationale for this change
In the week from 2026-10-29 to 2026-11-05, Comet CI used about 2,600
runner-hours on ASF runners. Pull requests were 1,333 of them, the merge queue
1,003, and the nightly run 243. Even after #7311, fork pull requests into
`main` used about 450 runner-hours in the 5.5 days after it landed. Nearly all
of that came from `run-*` labels. A fork run can't see the labels, so labeled
suites still run upstream, and because the labels stay on the pull request they
run again on every push. `run-spark-4.1-tests` and `run-iceberg-tests` are
among the most common, because without them a pull request gets no Spark SQL or
Iceberg verdict until the merge queue.
The PR tier now runs on the fork for fork pull requests, so Spark SQL 4.1
and Iceberg 1.11 can run there at no cost to ASF runners. The merge queue
already gates on exactly these versions. With this change a pull request gets
the queue's Spark and Iceberg verdict before it is queued, which removes the
main reason to apply those two labels, and it should also cut the queue runs
that fail on a 4.1 or 1.11 problem. In the same week, 174 runner-hours went to
failed queue runs and 154 to entries cancelled behind them.
## What changes are included in this PR?
- `dev/ci/compute-changes.py`:
- `spark_4_1` (catalyst and sql_core) and `iceberg_1_11` move from
`["queue", label]` to `["pr", "queue"]`. Path filters still apply.
- `spark_4_1_hive` stays queue-only, with `run-spark-4.1-hive-tests` to
bring it forward.
- Labels:
- `run-spark-4.1-tests` is retired. It gates nothing now, and a `labeled`
run for it starts nothing. The label can be deleted from repository settings
after merge.
- `run-iceberg-tests` now selects the three nightly Iceberg versions (1.8,
1.9, 1.10).
- `dev/ci/check-ci-config.py`: `PR_TIER`, `QUEUE_TIER`, `ICEBERG_OPT_IN` and
the label policy cases are updated to the new tiers.
- `.github/workflows/README.md`, the `spark_4_1` comment in `ci.yml`,
`docs/source/contributor-guide/ci.md` and `spark-sql-tests.md` are updated to
describe the new tiers.
Nothing else changes for the other tiers. The nightly run still covers Spark
SQL 3.5, 4.0 and 4.2, Iceberg 1.8, 1.9 and 1.10, the Delta suites and the other
Comet profiles. Pull requests that target a release branch already ran every
tier and are unaffected.
What this costs:
- Fork runs take longer, since the Spark SQL and Iceberg shards run
alongside the Linux test matrix.
- A pull request from a branch of this repository now pays for these two
suites upstream on each push when its paths match. That was 13 runs in the 5.5
days measured.
## How are these changes tested?
`python3 dev/ci/check-ci-config.py` passes. It evaluates `POLICY` against
the event cases in that file, which this PR updates: a plain pull request, a
fork push, a fork pull request, the merge queue, the nightly run, release
branches, and the remaining 4.1 hive and Iceberg label cases.
`test-iceberg-shards.py`, `test-fork-ci-status.cjs` and
`check-versioned-source-parity.py` also pass. This PR's own fork run is the
end-to-end check: it changes `dev/ci/**`, so it routes to the Spark SQL 4.1 and
Iceberg 1.11 suites on the fork.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
