Hi Shilun, Thanks for initiating this discussion and for all the efforts you're putting into the 3.5.0 release — much appreciated!
> Ongoing Support for the Hadoop 3.4 Branch It’s not just 3.4 — we currently have active release lines for 2.10, 3.2, and 3.3 as well. The officially designated EOL release lines are listed here [1]. Personally, I think it’s time we consider marking 3.2 and 3.3 as EOL. However, I would be cautious about doing the same for 3.4 immediately after the 3.5.0 release. 3.5.0 brings substantial changes, including the JDK migration, which might make immediate adoption difficult for some users. I suggest we keep 3.4.x as the sole active JDK 8-compatible line for a while longer. We could maintain it with selective cherry-picks for critical issues and only roll a release if there’s significant demand. We can always revisit this decision a few months after 3.5.0 is out. While many projects have moved to newer JDK versions, a few downstream consumers still rely on JDK 8 — that’s something we should factor in. > Preparation for Hadoop 3.5 Release Great to see progress on this front. My responses inline: > Should we set up a new CI pipeline for JDK 17 on the trunk branch? Yes, since we're bumping the compilation version to 17, our regular CI should move entirely to JDK 17. > Should we also add support for JDK 21 in CI? I don’t think running full test suites on both JDK 17 and 21 is practical. I propose we follow a similar approach as we did for JDK 11: As of the current code compilation targets JDK 17 only, the --release 17 flag is already hardcoded [2], so the generated bytecode is always JDK 17, regardless of whether you build with JDK 21, So we can configure pre-commit jobs via Yetus ensure multi-JDK compilation doesn’t break and a separate daily CI job can verify runtime compatibility on JDK 21 running tests. > If JDK 21 support is added, can we remove the existing JDK 11 pipeline? Yes, with 3.5.0, we can officially declare JDK 17+ as the baseline. JDK 11 can be dropped. And moreover if you bump the version at [3] to 17, JDK-11 won't compile only. > Operating System Support Strategy +1 to all three of your proposals. Maintaining outdated OS images is tedious and brings little value. It’s better to validate against newer, actively supported distros and ensure a smooth experience there. > Release Strategy Under Multi-JDK Builds I don't recall any active discussions around multi-JDK release variants. As mentioned earlier, we would be compiling with JDK 17 as the version at [3] would change to 17, and that should be the only binary variant we publish — i.e., hadoop-3.5.0 compiled with JDK 17. We can document how users can build from source with higher JDKs if needed. In essence, the official release is the source. Binaries are convenience artifacts — and should reflect our declared compilation baseline. The other sub-questions under this section become moot in light of the above, so skipping them for brevity. Just my thoughts — I’m open to adjusting the plan based on what the broader community agrees upon. Best, Ayush [1] https://cwiki.apache.org/confluence/display/HADOOP/EOL+%28End-of-life%29+Release+Branches [2] https://github.com/apache/hadoop/blob/729f19f233948b8472874cf7e2b65d3b33cd2ec6/hadoop-project/pom.xml#L2768-L2769 [3] https://github.com/apache/hadoop/blob/729f19f233948b8472874cf7e2b65d3b33cd2ec6/hadoop-project/pom.xml#L2204-L2205 On Fri, 11 Jul 2025 at 11:14, slfan1989 <slfan1...@apache.org> wrote: > > Dear community, > > I'd like to initiate a discussion on the future release and support strategy > for Hadoop. This includes a few important topics that I believe we should > align on: > > 1. Ongoing Support for the Hadoop 3.4 Branch > > As it stands, Hadoop 3.4 is the last release line that supports Java 8. It is > expected that Java 8 compilation support will be officially removed from the > trunk branch after a community vote. > At the same time, the trunk branch is increasingly diverging from the 3.4 > line due to the adoption of JUnit 5 and new features based on JDK17 syntax. > As a result, future bug fixes and backports will become significantly more > difficult. > Given this, I’d like to ask: after the release of hadoop-3.4.2, does the > community plan to continue maintaining and releasing hadoop-3.4.3 or other > 3.4-series versions? > > 2. Preparation for Hadoop 3.5 Release > > With the JDK17 support work nearing completion, it’s time to start planning > for the Hadoop 3.5 release. Specific questions for discussion include: > > Should we set up a new CI pipeline for JDK17 on the trunk branch? > > Should we also add support for JDK21 in CI? > > If JDK21 support is added, can we remove the existing JDK11 pipeline? > > 3. Operating System Support Strategy > > Several major OS distributions—CentOS 7, CentOS 8, and Debian 10—have already > reached or are approaching end-of-life (EOL). > Should we adjust our support policy for these systems? For example: > > Officially deprecate EOL systems; > > Provide documentation-only support for compilation; > > Remove outdated base images from Dockerfiles. > > 4. Release Strategy Under Multi-JDK Builds > > As we prepare to support JDK11, JDK17, and JDK21, we also need a clear > strategy for future releases. Key questions include: > > Should we publish multiple build variants like: hadoop-3.5.0-jdk11, > hadoop-3.5.0-jdk17, hadoop-3.5.0-jdk21? > > What naming convention should we adopt for these variants? > > For Maven Central artifacts, should we distinguish by JDK version or consider > a unified approach? > > Additional Context and Suggestions from My Side: > > The JDK17 compatibility work is being tracked under HADOOP-17177. I am in the > process of reviewing and linking all relevant JIRAs—both completed and > pending—to this umbrella JIRA, and will share an overview in the coming week. > > Regarding the JUnit 5 migration: > Based on our experience, JUnit 5 introduces significant differences compared > to JUnit 4. It fundamentally changes how tests are written and run. My > proposed upgrade path is as follows: > > Complete the migration to JUnit 5.8.2 (we are still working on HDFS and > Hadoop-Azure and aim to finish in the next two weeks); > > Remove JUnit 4 usage module by module (e.g., Yarn, MapReduce, HDFS, Common, > etc.); > > Upgrade to JUnit 5.13.2 (as Steve suggested); > > Upgrade the Maven Surefire Plugin. Earlier issues were due to the mixed use > of JUnit 4 and 5, which will no longer be a concern once JUnit 5 is fully > adopted; > > Upgrade the Maven version to 3.9.0 for better compatibility and plugin > support. > > Once the JUnit 5 migration is complete, we will conduct a thorough test audit > to ensure everything runs correctly and mark unstable tests for future > attention. > > Regarding legacy OS support, I recommend a "lightweight" approach: > > Remove unsupported base images from our Dockerfiles or upgrade them to newer > versions; > > Provide documentation to help users build on older systems; > > But formally discontinue full support for EOL systems. > > I’d appreciate hearing your thoughts on these topics. Looking forward to a > fruitful discussion! > > Best regards, > Shilun Fan. > > > On Wed, Jun 4, 2025 at 7:23 AM slfan1989 <slfan1...@apache.org> wrote: >> >> Hi, Chris >> >> Thank you very much for your efforts in driving the JDK17 upgrade. I will >> continue to follow up on the JUnit5 migration, and I hope we can >> successfully complete this upgrade together. >> >> Best Regards, >> Shilun Fan. >> >> On Mon, Mar 24, 2025 at 1:16 PM slfan1989 <slfan1...@apache.org> wrote: >> >> > Hello everyone, >> >> > I've mentioned in other forums that I'd volunteer as release manager for >> > Apache Hadoop 3.5.0, with a goal of full support for Java 17. I'd like to >> > get more specific about release planning and a potential target date. >> >> > It can be easy to get lost in the nested historical JIRAs for Java 17 >> > support [1]. I'd like to start by summarizing my understanding of work done >> > and work remaining for Java 17 as well as other significant features the >> > community wants to ship in 3.5.0: >> >> > - We've completed the Jersey upgrade [2]. >> > - We've made significant progress on the JUnit 5 upgrade [3], and we >> > have a list of several PRs remaining to be reviewed and committed [4]. >> > - There is some sun.misc usage that needs alternate implementations [5] >> > [6]. There are PRs, but they haven't had any recent activity. >> > - Once these issues are settled, we need to (re-)introduce a Java 17 >> > build profile [7]. >> > - I and a few others are also interested in trying to target the GCS >> > FileSystem implementation for 3.5.0 [8]. >> >> > Does this look accurate? Did I miss anything important for 3.5.0, either >> > for Java 17 or other features? >> >> > Chris Nauroth >> >> > [1] https://issues.apache.org/jira/browse/HADOOP-17177 >> > [2] https://issues.apache.org/jira/browse/HADOOP-15984 >> > [3] https://issues.apache.org/jira/browse/HADOOP-14693 >> > [4] https://github.com/apache/hadoop/pull/7337#issuecomment-2857464756 >> > [5] https://issues.apache.org/jira/browse/HADOOP-19329 >> > [6] https://issues.apache.org/jira/browse/HADOOP-19334 >> > [7] https://issues.apache.org/jira/browse/HADOOP-19298 >> > [8] https://issues.apache.org/jira/browse/HADOOP-19343 --------------------------------------------------------------------- To unsubscribe, e-mail: hdfs-dev-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-dev-h...@hadoop.apache.org
