Hi Dongjoon and community, +1 to dropping Java 17 in Spark 5.0.0.
Furthermore, I would like to propose considering Java 25 as the minimum version for Spark 5.0.0, dropping Java 21 as well. Making Java 25 the baseline brings substantial architectural and performance advantages: 1. G1 Region Pinning (JEP 423): Spark relies heavily on zstd-jni across shuffles, broadcast, caching, and storage. On Java 17 and 21, the lack of JEP 423 causes G1 to disable GC during JNI critical regions, easily triggering premature OOMs under default JVM settings (as noted in PR #51920). Java 25 includes JEP 423, which pins only the affected regions and resolves this issue. 2. Cleaner off-heap memory management (SPARK-53327): We currently carry a workaround for datasketches-memory to keep it working across Java 17, 21, and 25. Moving to Java 25+ allows us to upgrade to DataSketches 9.x, which uses the Foreign Function & Memory (FFM) API, and eliminate this workaround entirely. 3. Native data sources (SPARK-60101): Native data sources implemented in Rust or C++ rely on the FFM API, which finalized in Java 22. Java 21 users already cannot use native data sources without runtime errors. 4. SIMD-vectorized execution: Projects like Varka (https://github.com/vecbricks/varka) demonstrate significant performance gains for SQL execution by exploring SIMD vectorization that leverages modern JDK Vector API capabilities available in Java 25. 5. Ecosystem alignment: Other ecosystem projects are already exploring this transition; for instance, the Apache Hive community is actively discussing moving to Java 25+ (https://www.mail-archive.com/[email protected]/msg145376.html). 6. Lower migration friction via Spark Connect: The decoupled architecture of Spark Connect allows client applications running on older JDKs (Java 17 or 21) to communicate with a Java 25 Spark Connect server, greatly reducing the upgrade barrier for users. Given that Spark 5.0.0 is a major release and Spark 4.x continues to support Java 17 and 21, setting Java 25 as the baseline would allow us to eliminate legacy workarounds and fully capitalize on modern JVM features. Thanks, Cheng Pan > On Oct 29, 2026, at 06:44, Dongjoon Hyun <[email protected]> wrote: > > Hi, All. > > I'd like to propose dropping Java 17 support in Apache Spark 5.0.0, making > Java 21 > the minimum supported Java version. This is the same step we took in Apache > Spark > 4.0.0 for Java 8 and 11 (SPARK-44112). > > https://issues.apache.org/jira/browse/SPARK-60669 > https://github.com/apache/spark/pull/59897 > > Here are the reasons. > > 1. Java 21 is mature in our community. Apache Spark has officially supported > Java 21 > since 4.0.0 (May 2025), and our daily CI has tested Java 21 since June 2023 > (SPARK-43835). > > 2. Supporting both Java 17 and 21 has a maintenance cost. Since the PR > builder uses > Java 17, we keep separate Java 21 SQL golden files (`*.sql.out.java21`). > They have > become stale after merges and had to be regenerated separately several times > (SPARK-51937, SPARK-52400, SPARK-57560, SPARK-57649). We also keep Java > version-specific code paths and tests such as > `Utils.isJavaVersionAtLeast21`. > > 3. The ecosystem is moving on. For example, our Hive 4.2 metastore support > (SPARK-57263) requires Java 21, and newer dependency versions such as > Apache Derby 10.17 no longer support Java 17. > > 4. We can use Java 21 APIs and language features directly, e.g., virtual > threads > without reflection. > > This affects only the `master` branch (Apache Spark 5.0.0). `branch-4.x` > (Apache Spark 4.4.0) and the other 4.x branches keep Java 17 support, so Java > 17 > users can stay on the Apache Spark 4.x line. Apache Spark 5.0.0 will support > Java 21 and 25. > > SPARK-60669 changes the minimum Java version in the build, the default Java > version > in GitHub Actions, and the documentation. I'll clean up the remaining Java > 17-specific code, tests, CI images, and release scripts in separate > PRs. > > Please let me know if you have any concerns. > > Thanks, > Dongjoon Hyun. > > --------------------------------------------------------------------- > To unsubscribe e-mail: [email protected] >
