andygrove opened a new pull request, #6392: URL: https://github.com/apache/datafusion-comet/pull/6392
## Which issue does this PR close? Closes #6387. ## Rationale for this change About half of the merge-queue runs compile the native library from scratch: 76 of 155 successful full queue runs between 2026-09-11 and 2026-09-29 missed the cargo cache, and those took a median of 115 minutes against 88 for the rest. The repository's 10 GB cache budget cannot hold the entries main needs (about 12 GB, table in the issue), and because both cargo keys hashed the Rust sources, every Rust change on main wrote a fresh 5.5 to 6.7 GB pair. On 2026-09-29 the `cargo-ci` entry written at 06:18 was gone by 07:46. ## What changes are included in this PR? - `cargo-ci` is keyed on `Cargo.lock` and `Cargo.toml` only, in `pr_build_linux.yml`, `spark_sql_test_reusable.yml`, `iceberg_spark_test_reusable.yml` and `spark_sql_writer_tests.yml`, and it is saved only when that exact key is missing. That is one entry per dependency set instead of one per Rust change. The workspace crates recompile either way. - `linux-test-rust` no longer caches `native/target`. At 4 to 5 GB it was the largest entry, and the job is never the long pole of a run, so compiling its dependencies from scratch costs about 7 runner-minutes and no wall-clock time. With nothing to write, it now skips the push run (about 24 runner-minutes a push), and it drops out of `CACHE_REFRESH_JOBS`. Its Maven restore in the `rust-test` action now borrows the TPC jobs' `Linux-java-maven-*` entry, since an entry of its own would never be refreshed. - `docs.yaml` no longer uses `setup-java`'s `cache: 'maven'`. That entry was 2.5 GB, read by the docs deploy alone, which runs after the merge and gates nothing. - The README, `ci.md` and the `compute-changes.py` comment now describe three cache-writing jobs instead of four. The steady state on main should drop to about 6 GB: `cargo-ci` 1.4, `Linux-spark-sql` 2.0, `Linux-java-maven` 1.5, and the TPC datasets. That leaves room for the macOS cache in #6390. ## How are these changes tested? `python3 dev/ci/check-ci-config.py`, `actionlint` and prettier pass locally. This PR's own runs exercise the new restore key. The save path runs only on main, so the effect shows after the merge: `gh api repos/apache/datafusion-comet/actions/caches` should list a single `Linux-cargo-ci-<hash>` entry, and no `Linux-cargo-debug-*` or `setup-java-*-maven-*` entries once the old ones age out. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
