I suggest dropping Java 8 support first as it currently a blocker for parquet version bump https://github.com/apache/iceberg/pull/10209#issuecomment-2230161894
We can have a separate vote thread to drop Java 8 as this therad's title is to support Java 21. And IMO we should support Java 21 or latest after dropping Java 8 support to avoid consuming more CI resources by too many java builds. - Ajantha On Thu, Jul 11, 2024 at 3:21 PM Steve Loughran <ste...@cloudera.com.invalid> wrote: > A move to java 11 means it is time to move to Hadoop 3.3.x as the minimum > release, anything 17+ means java 3.4.x. Which before long will go making > java 11 its minimum version. > > That is: > - cut the hadoop2 version/profile which is really java7. > - be prepared to move to 3.4.x if some java11/17 incompatibility surfaces > as any issues there will only be on branch-3.4 and trunk branches. > > Cutting the hadoop2 profile can only be good as it reduces the test matrix > and means that if problems do surface they won't be rejected as WONTFIX + > /upgrade > > There are still some bits outstanding in the hadoop "support java 17" > JIRAs, but they should be all server-side and test issues, and *should* > be nothing to worry about. > https://issues.apache.org/jira/browse/HADOOP-17177 that is: filesystem > clients should all be good. And the CI tooling is getting set up for java17 > test runs > https://github.com/apache/hadoop/pull/6914 > > Meanwhile the first java 23 issue has surfaced: Oracle filed a bug report > warning that the whole Subject class is going to break so > UserGroupInformation.doAs() will go. > https://issues.apache.org/jira/browse/HADOOP-19212 > That's going to be traumatic across everything. Reflection is going to be > needed to use the successor classes/methods, but because the UGI calls are > common in hive and any other service which support different users in the > same process, getting adoption of the new operations will be tricking. > > steve > > > > > Java21 would be interesting; their new memory > > > On Thu, 11 Jul 2024 at 09:09, Gabor Kaszab <gaborkas...@apache.org> wrote: > >> Hey, >> >> About the part where we plan to drop the Hive module. This seems like a >> thing that has already been discussed and decided, so I might have missed >> that conversation. Would it make sense to involve Hive devs into this >> decision, though, so that we can make sure there is no difficulty on Hive >> side due to this? Mayba a vote (if we hadn't had it already)? >> >> Cheers, >> Gabor >> >> >> On Wed, Jul 10, 2024 at 5:34 PM Robert Stupp <sn...@snazy.de> wrote: >> >>> +1 on deprecating Java 8 in the next "catchable" Iceberg release. >>> >>> +1 on removing Java 8 in Iceberg 2.0 >>> >>> +1 on _building_ for Java 11 in Iceberg 2.0 >>> >>> +1 on _building_ with Java 17 in Iceberg 2.0 >>> >>> >>> I'd recommend to _build_ for Java 11 wherever possible (`javac --release >>> 11`), because many users are concerned about bumping 2 Java LTS versions (8 >>> ---> 17). >>> >>> >>> From our experience, it's fine to bump the JDK in CI to Java 17, that's >>> the Java version that works for Spark 3.3..3.5 (with some JPMS options). >>> I'm less concerned about Spark 4 requiring Java 17 (`--release 17` can >>> be configured for Spark 4, leaving everything at a lower version). >>> >>> I'm quite concerned about Avro 1.12 requiring Java 17 - maybe we can >>> stick w/ Avro 1.11.x. >>> >>> >>> On 10.07.24 15:13, Piotr Findeisen wrote: >>> >>> Hi, >>> >>> Thanks for additional feedback. It sounds like we're OK enabling builds >>> and testing with JDK 21 with the caveats that formatter is off. >>> The non-negotiable condition is that CI still checks format. >>> I think this is exactly what this PR >>> <https://github.com/apache/iceberg/pull/10474> is doing, which means we >>> maybe can merge it. One small step forward. >>> >>> BTW there is no risk of `./gradlew spotlessApply` producing different >>> results when being executed under JDK 21. >>> The spotless plugin needs to be fully disabled, and then this command >>> will fail saying the target is not found. >>> It would be even better to provide custom spotlessApply command stub >>> that would produce an actionable error message. >>> I will add something like this to the above PR. >>> >>> re dropping Java 8 >>> we need to figure out exact deprecation path and I don't know yet the >>> customs to drive this fully. >>> But happy to help as much as I can nonetheless. >>> >>> >>> Best >>> Piotr >>> >>> >>> >>> >>> >>> >>> On Wed, 10 Jul 2024 at 11:19, Eduard Tudenhöfner < >>> etudenhoef...@apache.org> wrote: >>> >>>> >>>>> It has a caveat (we can't run formatter on 21 and 8, and we need to >>>>>> choose one). >>>>> >>>>> >>>>> Would it format differently? I would go for 21 since that's the path >>>>> forward, but I'm also fine with JB's suggestion 👍 >>>>> >>>>>> >>>>>> >>>> Yeah it would format differently, because the underlying >>>> *google-java-format* picks a different formatting depending on the JDK >>>> version and that's why we had to pin the *google-java-format* version >>>> to the one that still supports JDK8. >>>> >>>> I think it's ok to disable the formatter when running with JDK 21 for >>>> people that want to use that JDK but we need to make sure that CI checks >>>> still run JDK < 21 in order to make sure code is properly formatted. >>>> >>>> The one risk I see is that it will be difficult to understand why >>>> running the build with JDK 21 passes locally but fails on CI (where an >>>> older JDK is used). >>>> Maybe it's sufficient to mention in the docs that *./gradlew >>>> spotlessApply* needs to be executed locally with an older JDK. >>>> >>>> Eduard >>>> >>> -- >>> Robert Stupp >>> @snazy >>> >>>
