> One thing I know is that it is necessary to upgrade JUnit 4 to JUnit 5
Do you know any specific reason or docs mentioning that? Just curious as to why we need to migrate to JUnit 5 for JDK-17, Looking here [1], it seems like it supports JDK-21 as well and even [2] mentions like it does support jdk-17 > By the way, there's a JIRA that we have had to revert twice > (HADOOP-19071)[1], which involves upgrading the maven-surefire-plugin. I am bit more curious on this one as well, the maven surefire doc [3] just mentions the minimum version for all the releases, does the latest one on 3.0.0-M* gives us a pass, IIRC 3.0.0-M4 is working for Hive JDK-17 (In Progress) and atleast that isn't creating a problem over there. Looking at this ticket [4], it seems like 3.0.0-M6 with Junit-4 works with JDK-17 or am I missing something? Can we attempt that version & unblock JDK-17 efforts, if it indeed blocks us? 3.0.0-M8 in future if we want to push for JDK-21[5] > Additionally, we are using Hadoop compiled with JDK 17 internally (mainly on > the current trunk branch with --add-opens=). We have replaced over 600 NM > instances, and this part is functioning normally. Therefore, I believe we are > essentially ready to upgrade to JDK 17. That sounds cool, that means you folks have already solved all the problems like Jersey + any other on trunk & it is just a matter of rebasing those fixes & just pushing it to the apache repo or is there some catch which I missed? -Ayush [1] https://github.com/junit-team/junit4/blob/main/.github/workflows/main.yml#L22 [2] https://github.com/junit-team/junit4/issues/1770 [3] https://maven.apache.org/surefire/maven-surefire-plugin/plugin-info.html [4] https://issues.apache.org/jira/browse/SUREFIRE-2007 [5] https://issues.apache.org/jira/browse/SUREFIRE-2135 On Tue, 30 Jul 2024 at 07:58, Ashutosh Gupta <ashutoshgupta...@gmail.com> wrote: > > Thanks, everyone, for starting this discussion. This sounds like a good plan > to start with. As I was working on the JUnit 4 to 5 upgrade, I paused for a > while as I got occupied with other stuff. But I would be happy to complete it > as part of the Java switch process. > > On Tue, Jul 30, 2024 at 4:21 AM slfan1989 <slfan1...@apache.org> wrote: >> >> Thank you very much for initiating this discussion! I am also very much >> looking forward to JDK 17. I have observed that using --add-opens= is quite >> common in other Apache projects. One thing I know is that it is necessary >> to upgrade JUnit 4 to JUnit 5. There are some JIRA issues currently being >> worked on for this. For YARN MapReduce, some pull requests have already >> been completed for this feature. Upgrading Jersey is a very challenging >> project. Currently, our upgrade strategy is to perform it all at once. If >> we can find a way to break Jersey down into individual modules for >> upgrading, it might be much easier. >> >> I have successfully debugged most of the unit tests for YARN's >> ResourceManager and NodeManager.(Upgrade Jersey 2.4.1) Additionally, we >> are using Hadoop compiled with JDK 17 internally (mainly on the current >> trunk branch with --add-opens=). We have replaced over 600 NM instances, >> and this part is functioning normally. Therefore, I believe we are >> essentially ready to upgrade to JDK 17. >> >> My idea is to create a separate branch specifically for upgrading to JDK >> 17. After debugging and ensuring everything works, we can then merge this >> branch into the trunk branch. >> >> By the way, there's a JIRA that we have had to revert twice >> (HADOOP-19071)[1], which involves upgrading the maven-surefire-plugin. >> >> [1] https://issues.apache.org/jira/browse/HADOOP-19071 >> >> - Shilun Fan >> >> On Tue, 30 Jul 2024 at 06:37, Shilun Fan wrote: >> >> > Depends on what we mean by switching to JDK-17, Compile time support >> > or Runtime support, We don't have compile time support for JDK-11 too, >> > it is just runtime, We have a daily build as well for JDK-11, which >> > has now some genuine failures now, need to check [1], as of now the >> > version is hardcoded here [2], unless we change here it but just >> > change our local JAVA_HOME, it will compile with JDK-17 as well I >> > believe >> >> > So, If we are to chase the runtime support we can setup a regular CI >> > with JDK-17 like we have one for JDK-11, there were some test failures >> > reported as part of HADOOP-18716, most of them should be fixable by >> > some dependency upgrades or some hacks like adding one of these >> > ``` >> > --add-opens=java.base/java.net=ALL-UNNAMED >> > --add-opens=java.base/java.util=ALL-UNNAMED >> > --add-opens=java.base/java.util.concurrent=ALL-UNNAMED >> > --add-opens=java.base/java.util.concurrent.atomic=ALL-UNNAMED >> > --add-opens=java.base/java.lang=ALL-UNNAMED >> > --add-opens=java.base/java.util.regex=ALL-UNNAMED >> > --add-opens=java.base/java.io=ALL-UNNAMED >> > ``` >> >> > That is some typical JDK-17 hack.... >> >> > For the compile support, that is we change the value at [2] to 17, I >> > think then some bigger problems will surface, One I know is Jersey >> > creates some issues for sure, We need to upgrade Jersey, As per this >> > [3] to Jersey 2.35+ or do some hack to use a patched jersey version >> > with minimal change which can unblock the JDK-17 initiative >> >> > -Ayush >> >> >> > [1] >> https://ci-hadoop.apache.org/job/hadoop-qbt-trunk-java11-linux-x86_64/709/ >> > [2] >> https://github.com/apache/hadoop/blob/038636a1b5250e06622cac7ee11b12965c91111e/hadoop-project/pom.xml#L160 >> > [3] https://github.com/eclipse-ee4j/jersey/wiki/Road-Map >> >> On Tue, 30 Jul 2024 at 00:53, PJ Fanning wrote: >> > >> > I think this is worth considering. I think it would require a minor >> > release like 3.5.0 as opposed to considering it for future 3.4.x patch >> > releases. >> > I tend to build locally with Java 11, by default and I haven't hit >> > major issues building Hadoop. There may be some gotcha somewhere but >> > it is likely to be easy enough to fix. >> > >> > It's probably only a matter of time before some important dependency >> > jars for Hadoop require Java 11 or 17. >> > >> > On Mon, 29 Jul 2024 at 20:04, Steve Loughran >> > wrote: >> > > >> > > A lot of projects are moving off java8. making java17 the new baseline >> > > >> > > what do we need to there that is blocker rather than just "nice"'? >> > >> > --------------------------------------------------------------------- >> > To unsubscribe, e-mail: hdfs-dev-unsubscr...@hadoop.apache.org >> > For additional commands, e-mail: hdfs-dev-h...@hadoop.apache.org >> > >> >> --------------------------------------------------------------------- >> To unsubscribe, e-mail: common-dev-unsubscr...@hadoop.apache.org >> For additional commands, e-mail: common-dev-h...@hadoop.apache.org --------------------------------------------------------------------- To unsubscribe, e-mail: hdfs-dev-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-dev-h...@hadoop.apache.org
