Hi, All. Following the previous discussion thread, "Drop Java 17 support in Apache Spark 5.0.0 (SPARK-60669)", I'd like to let you know that the `master` branch (Apache Spark 5.0.0) now requires Java 21 or later. Java 17 is no longer supported on `master`. Thank you to everyone who participated in the discussion and the reviews.
https://lists.apache.org/thread.html/3qqpjrc8k2k5sm657808v6slggz574dl This work is tracked under the umbrella JIRA issue, SPARK-60847 (Make Java 21 the minimum Java version). https://issues.apache.org/jira/browse/SPARK-60847 Resolved: - SPARK-60669 Drop Java 17 Support - SPARK-60831 Update BenchmarkBase to use Java 21 as the base version - SPARK-60833 Select old Spark versions by Java support in HiveExternalCatalogVersionsSuite - SPARK-60836 Support Spark versions like `3.5.10` in `HiveExternalCatalogVersionsSuite` In progress: - SPARK-52822 Upgrade Derby to 10.17.1.0 - SPARK-60848 Regenerate benchmark results after bumping the minimum Java to 21 Since this change affects all of us who work on `master`, here is a short summary. 1. What you may need to do - Please use JDK 21 or 25 to build `master`. Maven and SBT stop with an error on Java 17, so please check your `JAVA_HOME` and IDE settings. - If the GitHub Actions jobs of your open PR fail on Java 17, please rebase the PR onto the latest `master`. Our CI merges your PR branch into the latest `master`, but it runs with the Java version defined in the workflow files of your PR branch. - When you regenerate benchmark results, `*-results.txt` files are now for Java 21 and `*-jdk25-results.txt` files are for Java 25. There are no `*-jdk21-results.txt` files anymore. In addition, SPARK-60848 regenerates the benchmark results after the Java version bump. 2. What changes for users - All Apache Spark 5.0.0 artifacts are compiled for Java 21 (class file version 65). So, Java 21 is required on the client side as well as the cluster side, e.g., for the Spark Connect JVM client, the Spark Connect JDBC driver, `spark-launcher`, and libraries compiled against Apache Spark 5.0.0. The pure-Python Spark Connect client (`pyspark-client`) is not affected. - Existing release branches such as `branch-3.5` and `branch-4.*` are not affected and keep their Java support as-is. 3. Others - `HiveExternalCatalogVersionsSuite` used to skip all old Spark releases on Java 21 and later. Now it tests the old releases that support the running Java version (Apache Spark 4.0+ on Java 21 and 4.2+ on Java 25). So, `master` keeps its backward compatibility test without Java 17. - Dropping Java 17 also allows dependency upgrades that require Java 21. For example, SPARK-52822 upgrades Apache Derby to 10.17.1.0, which requires Java 21. This is the last release of Apache Derby because the Derby developers voted to retire the project in October 2025. Since this is a big change, I'd like to ask for your help. Please try the latest `master` with Java 21 (and Java 25 if possible) in your environments and downstream projects. If you hit any issue, e.g., build or test failures, leftover Java 17 references in docs or scripts, or anything unexpected, please file a JIRA issue under SPARK-60847 or reply to this thread. A few remaining cleanups will be tracked there as well. Thank you in advance. Dongjoon. --------------------------------------------------------------------- To unsubscribe e-mail: [email protected]
