manuzhang opened a new pull request, #5897:
URL: https://github.com/apache/datafusion-comet/pull/5897
## Which issue does this PR close?
Closes #4844.
## Rationale for this change
JDK 11 support was deprecated in 1.0.0 (#4857), with removal scheduled for
1.1.0.
Spark 3.4 and 3.5 themselves still run on JDK 11, as noted on #4844.
However, Comet's published artifacts already require JDK 17:
- `dev/release/build-release-comet.sh` refuses to build on anything older
than JDK 17.
- On JDK 17+, the `jdk17` profile is declared after the Spark profiles, so
it overrode their `java.version=11`.
- As a result, the 1.0.0 `comet-spark-spark3.4_2.12` and
`comet-spark-spark3.5_2.12` jars contain 610 Comet classes with class-file
major version 61 (Java 17), including `NativeBase` and
`CometShuffleBlockIterator`. Those jars cannot load on JDK 11 today.
JDK 11 only worked when building Comet from source on JDK 11 for Spark 3.x.
This PR makes the JDK 17 requirement explicit and drops the JDK 11 CI coverage.
## What changes are included in this PR?
**Build**
- `pom.xml`: the default `java.version` and the `spark-3.4` / `spark-3.5`
profiles now target 17. The `jdk11` and `jdk17` profiles are removed; with 17
as the default, `jdk17` would be a no-op. Passing `-Pjdk17` now only logs
Maven's "profile does not exist" warning.
- A new `require-java-17` enforcer execution fails the `validate` phase on
older JDKs with `Comet requires JDK 17 or later to build.`, rather than a later
compiler error.
**CI**
- `ci.yml`: Preflight (RAT check) uses JDK 17. The Spark 3.4 Spark SQL tests
and Iceberg 1.8 (Spark 3.4.3) run on JDK 17.
- `pr_build_linux.yml`: the Spark 3.4 lint and test profiles move to JDK 17.
`JAVA_TOOL_OPTIONS` is no longer conditional, since every profile is now JDK 17
or 21. The job names change from `Spark 3.4, JDK 11, Scala 2.12` to `Spark 3.4,
JDK 17, Scala 2.12`. Nothing in `.asf.yaml` or `dev/ci` refers to the old
names, because merges are gated on the `Required Checks` aggregator.
**Docs**
- `installation.md` and `compatibility/spark-versions.md`: Spark 3.4 and 3.5
are listed with Java 17. The JDK 11 deprecation warning is replaced by a
statement that JDK 17 or later is required. The Spark 3.4 deprecation warning
is unchanged.
- Contributor guide:
- `iceberg-spark-tests.md`: all Iceberg jobs now run on Java 17.
- `development.md`: the `-Pjdk17` guidance is removed.
- `benchmarking_micro_ec2.md`: the `Class java.lang.Record not found`
troubleshooting entry, which only happened on JDK < 17, is replaced by the new
enforcer message.
**Not changed**
- `kubernetes.md` still references an `apache/datafusion-comet:...-java11`
image tag. No workflow in this repo publishes Comet Docker images anymore, so I
did not switch it to a tag that may not exist.
- The remaining "JDK 11+" mentions describe JFR or JDK behavior, not Comet
support.
#5885 edits the same Spark 3.4 warning lines in `installation.md` and
`spark-versions.md`, so whichever PR lands second will need a small rebase.
## How are these changes tested?
- `./mvnw help:effective-pom` on JDK 21: `maven.compiler.source` and
`maven.compiler.target` are both 17 with no profile, `-Pspark-3.4`,
`-Pspark-3.5` and `-Pspark-4.0`.
- `./mvnw -N validate` on JDK 21: the `require-java-17` rule passes. I did
not check the failure path on JDK 11, because no JDK 11 was available locally.
- `./mvnw -Pspark-3.4 -Pscala-2.12 -pl common,spark -am -DskipTests
test-compile` on JDK 21 succeeds, and the enforcer rule runs in each module.
- `dev/ci/check-ci-config.py`, `actionlint` and `prettier --check` pass on
the changed files.
Running Spark 3.4 on JDK 17 in the Spark SQL and Iceberg 1.8 jobs is only
exercised in the merge queue or with the `run-spark-3.4-tests` /
`run-iceberg-tests` labels. It would be good to add those labels to this PR.
🤖 Generated with [Claude Code](https://claude.com/claude-code)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
