Thank you very much for initiating this discussion! I am also very much looking forward to JDK 17. I have observed that using --add-opens= is quite common in other Apache projects. One thing I know is that it is necessary to upgrade JUnit 4 to JUnit 5. There are some JIRA issues currently being worked on for this. For YARN MapReduce, some pull requests have already been completed for this feature. Upgrading Jersey is a very challenging project. Currently, our upgrade strategy is to perform it all at once. If we can find a way to break Jersey down into individual modules for upgrading, it might be much easier.
I have successfully debugged most of the unit tests for YARN's ResourceManager and NodeManager.(Upgrade Jersey 2.4.1) Additionally, we are using Hadoop compiled with JDK 17 internally (mainly on the current trunk branch with --add-opens=). We have replaced over 600 NM instances, and this part is functioning normally. Therefore, I believe we are essentially ready to upgrade to JDK 17. My idea is to create a separate branch specifically for upgrading to JDK 17. After debugging and ensuring everything works, we can then merge this branch into the trunk branch. By the way, there's a JIRA that we have had to revert twice (HADOOP-19071)[1], which involves upgrading the maven-surefire-plugin. [1] https://issues.apache.org/jira/browse/HADOOP-19071 - Shilun Fan On Tue, 30 Jul 2024 at 06:37, Shilun Fan wrote: > Depends on what we mean by switching to JDK-17, Compile time support > or Runtime support, We don't have compile time support for JDK-11 too, > it is just runtime, We have a daily build as well for JDK-11, which > has now some genuine failures now, need to check [1], as of now the > version is hardcoded here [2], unless we change here it but just > change our local JAVA_HOME, it will compile with JDK-17 as well I > believe > So, If we are to chase the runtime support we can setup a regular CI > with JDK-17 like we have one for JDK-11, there were some test failures > reported as part of HADOOP-18716, most of them should be fixable by > some dependency upgrades or some hacks like adding one of these > ``` > --add-opens=java.base/java.net=ALL-UNNAMED > --add-opens=java.base/java.util=ALL-UNNAMED > --add-opens=java.base/java.util.concurrent=ALL-UNNAMED > --add-opens=java.base/java.util.concurrent.atomic=ALL-UNNAMED > --add-opens=java.base/java.lang=ALL-UNNAMED > --add-opens=java.base/java.util.regex=ALL-UNNAMED > --add-opens=java.base/java.io=ALL-UNNAMED > ``` > That is some typical JDK-17 hack.... > For the compile support, that is we change the value at [2] to 17, I > think then some bigger problems will surface, One I know is Jersey > creates some issues for sure, We need to upgrade Jersey, As per this > [3] to Jersey 2.35+ or do some hack to use a patched jersey version > with minimal change which can unblock the JDK-17 initiative > -Ayush > [1] https://ci-hadoop.apache.org/job/hadoop-qbt-trunk-java11-linux-x86_64/709/ > [2] https://github.com/apache/hadoop/blob/038636a1b5250e06622cac7ee11b12965c91111e/hadoop-project/pom.xml#L160 > [3] https://github.com/eclipse-ee4j/jersey/wiki/Road-Map On Tue, 30 Jul 2024 at 00:53, PJ Fanning wrote: > > I think this is worth considering. I think it would require a minor > release like 3.5.0 as opposed to considering it for future 3.4.x patch > releases. > I tend to build locally with Java 11, by default and I haven't hit > major issues building Hadoop. There may be some gotcha somewhere but > it is likely to be easy enough to fix. > > It's probably only a matter of time before some important dependency > jars for Hadoop require Java 11 or 17. > > On Mon, 29 Jul 2024 at 20:04, Steve Loughran > wrote: > > > > A lot of projects are moving off java8. making java17 the new baseline > > > > what do we need to there that is blocker rather than just "nice"'? > > --------------------------------------------------------------------- > To unsubscribe, e-mail: hdfs-dev-unsubscr...@hadoop.apache.org > For additional commands, e-mail: hdfs-dev-h...@hadoop.apache.org > --------------------------------------------------------------------- To unsubscribe, e-mail: common-dev-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-dev-h...@hadoop.apache.org
