andygrove opened a new pull request, #5902:
URL: https://github.com/apache/datafusion-comet/pull/5902
## Which issue does this PR close?
Closes #5899.
## Rationale for this change
Anyone who wants to try a fix or feature before the next release currently
has to build Comet from source, including the Rust native library. Publishing
nightly SNAPSHOT jars to the ASF snapshot repository gives users and downstream
projects a ready-made artifact for the current development version, the same
way Spark and Iceberg publish their snapshots.
The repository previously had a tag-triggered GHCR Docker publish workflow,
removed in #4241. Its runs had failed for months, at first within a minute of
starting and later as `startup_failure` from unpinned third-party actions, and
it built arm64 under QEMU with caching disabled. This workflow avoids all three
problems: it only uses `actions/*` and the repo's own composite actions, builds
each architecture on a native runner, and caches the Cargo registry and Maven
repository.
## What changes are included in this PR?
A new scheduled workflow, `.github/workflows/publish_snapshot.yml`, with
three jobs:
- `changes` skips the nightly run when `main` has had no commits since the
previous night, so Nexus does not accumulate identical snapshots. A manual
dispatch always runs.
- `native` builds `libcomet.so` for linux/amd64 on `ubuntu-24.04` and
linux/aarch64 on `ubuntu-24.04-arm`. Both build inside an `ubuntu:20.04`
container so the library links against glibc 2.31, the same baseline as the
release builder in `dev/release/comet-rm/Dockerfile`, and a step fails the job
if the library ever requires a newer glibc. The same `make core-*-libs` targets
as the release are used, so the baseline CPU targets match released jars.
- `deploy` places both libraries under
`spark/target/classes/org/apache/comet/linux/` the way `build-release-comet.sh`
does, then runs `./mvnw deploy` for the four default variants: Spark 3.4 and
3.5 with Scala 2.12, Spark 4.0 and 4.1 with Scala 2.13. Spark 3.4 is built with
JDK 11 and the rest with JDK 17, matching `pr_build_linux.yml`.
`-Dmaven.deploy.skip=false` overrides the root pom so the parent pom is
deployed too, since consumers need it to resolve the child poms and the release
publishes it. Before deploying, every jar is checked to contain both native
libraries.
Credentials come from the `NEXUS_USER` and `NEXUS_PW` repository secrets
that ASF Infra provisions for snapshot publishing, read into a `settings.xml`
through `${env.*}` rather than written to disk. The `org.apache:apache` parent
pom already maps SNAPSHOT deploys to `apache.snapshots.https`, so no pom
changes are needed. If the secrets are not yet configured on this repository,
the deploy step fails at upload and an INFRA ticket is needed.
A `dry_run` input on `workflow_dispatch` runs the whole pipeline but ends
with `install` instead of `deploy` and uploads the jars as workflow artifacts.
It is also allowed on forks so the workflow can be exercised before a change
lands.
The installation guide's snapshot-only section now explains where the
snapshots live, which artifacts exist, that they are unreleased builds for
testing only, and how to use one with `spark-shell` either by downloading the
jar or via `--packages` with the snapshot repository. The workflows README
lists the new standalone workflow.
## How are these changes tested?
- `actionlint`, `prettier --check` and `dev/ci/check-ci-config.py` pass
locally.
- A local `./mvnw deploy -Dmaven.deploy.skip=false -Pspark-4.1
-DaltDeploymentRepository=...` into a file repository confirmed that the parent
pom, `comet-common`, `comet-spark` and `comet-spark-integration` are all
deployed with timestamped snapshot names.
- A `dry_run` of the workflow on my fork built both native libraries in the
Ubuntu 20.04 containers, passed the glibc check, built all four variants and
verified that each jar bundles both libraries:
https://github.com/andygrove/datafusion-comet/actions/runs/34770970719
Cost per night from that run: each native build takes about 17 minutes on
its runner, and the deploy job about 15 minutes for the four Maven builds, so
roughly 50 runner-minutes and 32 minutes of wall clock. The Cargo registry and
Maven caches were cold on the fork.
The first real publish will be a manual `workflow_dispatch` after this
merges, followed by checking that the coordinates resolve from the snapshot
repository.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
