sunchao opened a new pull request, #5841: URL: https://github.com/apache/datafusion-comet/pull/5841
## Which issue does this PR close? Part of #5830. Follow-up to #3249, which shared native artifacts within individual workflows. ## Rationale for this change The umbrella CI workflow still compiles the same default Linux native library separately for Linux, each selected Spark version, and each selected Iceberg version. A source-change PR normally builds four copies; a full main-branch run builds nine. This change builds one copy and shares it with those consumers, reducing duplicate compilation and runner usage. Actual time savings depend on cache state and will need measurement in CI. ## What changes are included in this PR? - Add one reusable Linux native producer using the existing Cargo `ci` profile, JDK 17, compiler flags, and main-only cache save policy. - Make Linux, Spark SQL, and Iceberg callers wait for that producer and pass its artifact through a required input. Keep Spark/JDK-specific compiled JVM artifacts separate. - Preserve existing path, event, and opt-in label selection. The producer runs whenever any consumer is selected, including Spark-only changes. - Remove duplicate native compilation and uploads from consumers. Keep Iceberg shard preparation as a small separate job. - Add configuration guards and regression coverage for producer selection, caller dependencies, artifact mapping, and accidental duplicate producers. The Linux reusable workflow, including lint and Rust debug tests, now starts after the shared native build. Rust formatting also runs before native compilation. macOS, Rust debug tests, and feature-specific workflows retain their separate builds. Artifact retention remains one day. ## How are these changes tested? Passed locally: - `actionlint -color -shellcheck=` - `python3 dev/ci/check-ci-config.py` - `python3 dev/ci/test-ci-config.py` (17 tests) - `python3 dev/ci/test-native-build-selection.py` (10 tests, including all 512 change-flag combinations across event/label cases) - `python3 dev/ci/check-suites.py` - `python3 dev/ci/test-iceberg-shards.py` (15 tests) and shard-matrix generation - Prettier check on the changed Markdown, Apache RAT, and `git diff --check` This is a workflow-only change; no full native or Spark build was run locally. Hosted CI still needs to validate artifact transfers and consumer execution, including opt-in Spark 3.4/JDK 11. Failed-job reruns can reuse the successful producer's artifact while it is retained; after expiry, rerun the full workflow. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
