+1 to Dongjoon's proposal. W.r.t. Cheng Pan's proposal, for sure I see clear benefit in moving to Java 25, but from the ecosystem alignment perspective, the general approach has been the case that we take conservative path rather than being adventurous.
FFM API is the major loss but is it that there is an API which is only available in Java 22? (It doesn't look to be a case since I assume we build the master branch with Java 21.) If it is not the case, for some users who want to use some newer features could run with Java 25, while other users wanting to stay in Java 21 can stay. That doesn't block the majority of existing Spark features. SIMD-vectorized execution on Java would be an interesting one, but it is not urgent since it has been years since 3rd parties just took a further step and built the native engine. It isn't to protect vendors - there are open sources like Apache DataFusion Comet doing this, so it's for free already. Actually, it is too late to internalize. (I see the value of internalization, but if it can't still match with native engine, it will still go that way.) On Fri, Oct 30, 2026 at 1:00 AM Xiao Li <[email protected]> wrote: > +1 to Dongjoon's proposal. I support Java 21 as the minimum version for > Spark 5.0. I do not support Java 25 as the minimum. > > Java 25 has useful improvements, including G1 region pinning and FFM > support. These are good reasons to support Java 25, but not to require it. > > Java 21 is a mature LTS release. Many enterprise users need more time to > upgrade their Java environments, libraries, and infrastructure. If we > require Java 25, we will make the Spark 5.0 upgrade more difficult for > these users. This could slow Spark 5.0 adoption. > > We should also keep the minimum supported Java version separate from the > recommended version. Users who want Java 25 features can use Java 25. Other > users should still be able to use Java 21. > > Dropping Java 17 already reduces our maintenance work. Java 21 gives us a > good balance between new features and compatibility. > > My preference is to make Java 21 the minimum for Spark 5.0, fully support > and test Java 25, and review the minimum version again for a future major > release. > > Thanks, > > Xiao > > Dongjoon Hyun <[email protected]> 于2026年10月28日周三 15:44写道： > >> Hi, All. >> >> I'd like to propose dropping Java 17 support in Apache Spark 5.0.0, >> making Java 21 >> the minimum supported Java version. This is the same step we took in >> Apache Spark >> 4.0.0 for Java 8 and 11 (SPARK-44112). >> >> https://issues.apache.org/jira/browse/SPARK-60669 >> https://github.com/apache/spark/pull/59897 >> >> Here are the reasons. >> >> 1. Java 21 is mature in our community. Apache Spark has officially >> supported Java 21 >> since 4.0.0 (May 2025), and our daily CI has tested Java 21 since June >> 2023 >> (SPARK-43835). >> >> 2. Supporting both Java 17 and 21 has a maintenance cost. Since the PR >> builder uses >> Java 17, we keep separate Java 21 SQL golden files >> (`*.sql.out.java21`). They have >> become stale after merges and had to be regenerated separately several >> times >> (SPARK-51937, SPARK-52400, SPARK-57560, SPARK-57649). We also keep Java >> version-specific code paths and tests such as >> `Utils.isJavaVersionAtLeast21`. >> >> 3. The ecosystem is moving on. For example, our Hive 4.2 metastore support >> (SPARK-57263) requires Java 21, and newer dependency versions such as >> Apache Derby 10.17 no longer support Java 17. >> >> 4. We can use Java 21 APIs and language features directly, e.g., virtual >> threads >> without reflection. >> >> This affects only the `master` branch (Apache Spark 5.0.0). `branch-4.x` >> (Apache Spark 4.4.0) and the other 4.x branches keep Java 17 support, so >> Java 17 >> users can stay on the Apache Spark 4.x line. Apache Spark 5.0.0 will >> support >> Java 21 and 25. >> >> SPARK-60669 changes the minimum Java version in the build, the default >> Java version >> in GitHub Actions, and the documentation. I'll clean up the remaining >> Java 17-specific code, tests, CI images, and release scripts in separate >> PRs. >> >> Please let me know if you have any concerns. >> >> Thanks, >> Dongjoon Hyun. >> >> --------------------------------------------------------------------- >> To unsubscribe e-mail: [email protected] >> >>
