andygrove opened a new pull request, #5963: URL: https://github.com/apache/datafusion-comet/pull/5963
## Which issue does this PR close? Follow-on to #5870 and #5938. Stacked on #5939: this PR shows that PR's three commits until it merges, and only the last commit is new here. ## Rationale for this change Every merge-queue run currently executes three Spark SQL versions, four Iceberg versions and the Linux test matrix against all five Spark profiles. Sampling the last six queue runs on 2026-09-15: | Job group | runner-min per queue run | | ------------------------------- | -----------------------: | | PR Build (Linux), all profiles | 470 | | Spark SQL 4.0 | 273 | | Spark SQL 4.1 | 267 | | Spark SQL 3.5 | 208 | | Iceberg 1.11 | 166 | | Iceberg 1.10 | 138 | | Iceberg 1.9 | 133 | | macOS build | 124 | | Iceberg 1.8 | 122 | | PyArrow, Delta, benchmark, lint | 22 | | Total | 1923 | Spark 3.5, Spark 4.0 and Iceberg 1.8/1.9/1.10 are about 870 of those minutes, roughly 45% of every queue run, and the analysis in #5870 found the old Iceberg versions were the most common reason the queue tier went red on a tree that was fine. The regressions those suites catch are real but rare, and a day's delay in seeing one costs less than paying for the suites on every merge. ## What changes are included in this PR? - A `nightly` tier in `POLICY` (`dev/ci/compute-changes.py`), fired by a `schedule` trigger on `ci.yml` at 06:00 UTC. On that event every path filter counts as matched and only `nightly` jobs run; the queue already ran everything else against the same tree. - The queue keeps one Spark version (4.1, the default build profile, all modules) and one Iceberg version (1.11, the only Spark 4.1 coverage). Spark SQL on 3.5 and 4.0, Iceberg 1.8/1.9/1.10 and the `linux-test` matrix's non-default Spark profiles move to the nightly tier. Their existing `run-*` labels still opt a pull request in early. - The nightly run skips itself when `HEAD` is older than the schedule interval plus 30 minutes, the same rule `publish_snapshot.yml` uses, so a quiet weekend does not re-test the same tree. - A `nightly_report` job, on the scheduled event only and only when `required_checks` is not green, opens an issue labelled `ci-nightly-failure` linking the run and listing the failed jobs, or comments on the one already open so consecutive red nights accumulate in one place. The label exists in the repository. - The nightly run gets its own concurrency subgroup, since it shares the tip-of-main sha with that commit's push run. - The `linux-test` profile selector `queue-only` is renamed `nightly`, and the profile rows' tier with it, since those rows no longer run in the queue. - `check-ci-config.py` gains a `NIGHTLY_TIER` set, subtracts it from `QUEUE_TIER`, and adds a `schedule` case, so a suite drifting back into the queue fails preflight. `nightly_report` is exempt from `required_checks.needs` for the same reason `docs` is. - The workflows README, the CI contributor-guide page (three tiers, a "Nightly runs" section), and the Spark SQL, Iceberg and development pages are updated to describe the three tiers. ## How are these changes tested? - `python3 dev/ci/check-ci-config.py` passes, including the new `schedule` case and the `merge_group` case now asserting the nightly suites are absent. - `EVENT_NAME=schedule python3 dev/ci/compute-changes.py /dev/null` reports exactly `build_linux_all_profiles`, `spark_3_5`, `spark_4_0`, `iceberg_1_8`, `iceberg_1_9` and `iceberg_1_10` true; with `NIGHTLY_STALE=true` everything is false. - `python3 dev/ci/linux-test-profiles.py --profiles nightly` returns the four non-default profile rows. - `actionlint --shellcheck=off` and `prettier --check` pass on the edited files. - The `nightly_report` job cannot run before this lands, since `schedule` only fires from the default branch. I will dispatch `ci.yml` and watch the first scheduled run after merge. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
