Please try Apache Spark 3.3+ (SPARK-33772) with Java 17 on your cluster simply, Jason.
I believe you can set up for your Spark 3.3+ jobs to run with Java 17 while your cluster(DataNode/NameNode/ResourceManager/NodeManager) is still sitting on Java 8. Dongjoon. On Fri, Dec 8, 2023 at 11:12 PM Jason Xu <jasonxu.sp...@gmail.com> wrote: > Dongjoon, thank you for the fast response! > > Apache Spark 4.0.0 depends on only Apache Hadoop client library. > > To better understand your answer, does that mean a Spark application built > with Java 17 can successfully run on a Hadoop cluster on version 3.3 and > Java 8 runtime? > > On Fri, Dec 8, 2023 at 4:33 PM Dongjoon Hyun <dongj...@apache.org> wrote: > >> Hi, Jason. >> >> Apache Spark 4.0.0 depends on only Apache Hadoop client library. >> >> You can track all `Apache Spark 4` activities including Hadoop dependency >> here. >> >> https://issues.apache.org/jira/browse/SPARK-44111 >> (Prepare Apache Spark 4.0.0) >> >> According to the release history, the original suggested timeline was >> June, 2024. >> - Spark 1: 2014.05 (1.0.0) ~ 2016.11 (1.6.3) >> - Spark 2: 2016.07 (2.0.0) ~ 2021.05 (2.4.8) >> - Spark 3: 2020.06 (3.0.0) ~ 2026.xx (3.5.x) >> - Spark 4: 2024.06 (4.0.0, NEW) >> >> Thanks, >> Dongjoon. >> >> On 2023/12/08 23:50:15 Jason Xu wrote: >> > Hi Spark devs, >> > >> > According to the Spark 3.5 release notes, Spark 4 will no longer support >> > Java 8 and 11 (link >> > < >> https://spark.apache.org/releases/spark-release-3-5-0.html#upcoming-removal >> > >> > ). >> > >> > My company is using Spark on Yarn with Java 8 now. When considering a >> > future upgrade to Spark 4, one issue we face is that the latest version >> of >> > Hadoop (3.3) does not yet support Java 17. There is an open ticket ( >> > HADOOP-17177 <https://issues.apache.org/jira/browse/HADOOP-17177>) for >> this >> > issue, which has been open for over two years. >> > >> > My question is: Does the release of Spark 4 depend on the availability >> of >> > Java 17 support in Hadoop? Additionally, do we have a rough estimate for >> > the release of Spark 4? Thanks! >> > >> > >> > Cheers, >> > >> > Jason Xu >> > >> >> --------------------------------------------------------------------- >> To unsubscribe e-mail: dev-unsubscr...@spark.apache.org >> >>
