Re: Building with JDK 21

Thanks Piotr for driving this, late +1 to add JDK 21 support and your plan for spotless. It seems ok to me too to bite the bullet and move to newer spotless (disabling spotless for JDK8 builds) post 1.6, but looks like the discussion happened and I'm fine either way. Thanks! Szehon On Mon, Jul 2

Re: Building with JDK 21

Thanks Fokko. I like the idea. Started a new "Dropping JDK 8 support" thread to ensure transparency. Best Piotr On Mon, 22 Jul 2024 at 15:24, Fokko Driesprong wrote: > Thanks for summarizing this, Piotr. > > I believe having a separate thread on dropping Java 8 is the right thing > to do. We w

Re: Building with JDK 21

Thanks for summarizing this, Piotr. I believe having a separate thread on dropping Java 8 is the right thing to do. We want to be as transparent about these changes as possible. Kind regards, Fokko Driesprong Op ma 22 jul 2024 om 14:37 schreef Piotr Findeisen < piotr.findei...@gmail.com>: > Tha

Re: Building with JDK 21

Thanks for this lively discussion, it is great to see so many great people involved! We have unanimous agreement that we add support for JDK 21. Partial support (without spotless) will be added after 1.6.0 release is out (just not to mess up with the release). Full support (with spotless) will be

Re: Building with JDK 21

+1 for dropping JDK8 support and adding JDK21. > What does dropping Java 8 support mean to companies that are still using Java 8 for Iceberg in production? >From the AWS side, AWS Corretto JDK8 end of life is July 2026, see: https://aws.amazon.com/corretto/faqs/#support_calendar. I would suggest

Re: Building with JDK 21

+1 adding java 21 support +1 removing java 8 support On Fri, Jul 19, 2024 at 1:33 PM Daniel Weeks wrote: > I'm also in favor of removing Java 8 support. Hive docs state Hive 3 > requires java 8 and > in prior cases there were potential co

Re: Building with JDK 21

I'm also in favor of removing Java 8 support. Hive docs state Hive 3 requires java 8 and in prior cases there were potential correctness issues when running with newer Java versions (these may have been addressed). As long as we're not upda

Re: Building with JDK 21

I agree that if we can separate the discussion about how to support Hive, then we should do that. +1 to removing Java 8 support +1 to adding Java 21 support. On Fri, Jul 19, 2024 at 12:58 PM huaxin gao wrote: > +1 in favor of adding java 21 support > +1 in favor of removing java 8 support > > I

Re: Building with JDK 21

+1 in favor of adding java 21 support +1 in favor of removing java 8 support I am currently working on Spark 4.0 / Iceberg integration . Spark 4.0 runs on Java 17/21. On Fri, Jul 19, 2024 at 4:58 AM Piotr Findeisen wrote: > Hi, > > We recently start

Re: Building with JDK 21

Hi, We recently started to test Hive3 with Java 11 and 17 and the tests pass. So dropping Java 8 doesn't technically require removing the Hive 3 related modules, unless users cannot do anything useful with them (because e.g. they can only run Hive run

Re: Building with JDK 21

Back to the main topic: Removing java8 support still allows us to use the old HMS libraries, so iceberg-hive-metastore module is not affected in this sense. We still need to run the tests, to ensure that there is no changes in the behaviour, but it is unlikely. +1 in favor of adding java 21 suppo

Re: Building with JDK 21

A basic question, is iceberg-hive-metastore considered part of the "Hive module"? I suppose that HMS 2.x is still widely used. AFAIK, the current iceberg-hive-metastore is compatible with HMS 2.1+, based on Iceberg and Spark CI, I also suppose it works well with Java 8 to 21. Thanks, Cheng Pan O

Re: Building with JDK 21

Thanks for the context, Denys and Peter. Sounds like there's a good question here about where the Hive integration should live and the most recent decision was to maintain that support in Hive. I definitely hear the point about Hive 3 users depending on the Iceberg modules. I'm also glad to hear th

Re: Building with JDK 21

In the following 1-2 months we plan to release HIVE-4.0.1 which includes bug fixes and then focus on HIVE-4.1.0 release with jdk17.

Re: Building with JDK 21

Hi Team, I was there when we decided to move the Hive development from the Iceberg repo to the Hive repo. Apart from the previously mentioned reasons there was another big blocker which prevented us moving forward with the development in the Iceberg repo - the lack of Hive releases. We needed sever

Re: Building with JDK 21

Hi All, Let me chime in here and add some Hive perspective on that. The only reason Iceberg support has moved into Hive, is that we didn't get enough support from the existing community. Our PRs got stuck pending review and even if we got some +1 those were not binding. Don't take me wrong, it

Re: Building with JDK 21

About the part where we plan to drop the Hive module. This seems like a thing that has already been discussed and decided, so I might have missed that conversation. I don’t remember saying that we plan to drop the Hive module, but given that Iceberg support has moved into Hive I think that we’ve b

Re: Building with JDK 21

I suggest dropping Java 8 support first as it currently a blocker for parquet version bump https://github.com/apache/iceberg/pull/10209#issuecomment-2230161894 We can have a separate vote thread to drop Java 8 as this therad's title is to support Java 21. And IMO we should support Java 21 or late

Re: Building with JDK 21

A move to java 11 means it is time to move to Hadoop 3.3.x as the minimum release, anything 17+ means java 3.4.x. Which before long will go making java 11 its minimum version. That is: - cut the hadoop2 version/profile which is really java7. - be prepared to move to 3.4.x if some java11/17 incompa

Re: Building with JDK 21

Hey, About the part where we plan to drop the Hive module. This seems like a thing that has already been discussed and decided, so I might have missed that conversation. Would it make sense to involve Hive devs into this decision, though, so that we can make sure there is no difficulty on Hive sid

Re: Building with JDK 21

+1 on deprecating Java 8 in the next "catchable" Iceberg release. +1 on removing Java 8 in Iceberg 2.0 +1 on _building_ for Java 11 in Iceberg 2.0 +1 on _building_ with Java 17 in Iceberg 2.0 I'd recommend to _build_ for Java 11 wherever possible (`javac --release 11`), because many users ar

Re: Building with JDK 21

Hi, Thanks for additional feedback. It sounds like we're OK enabling builds and testing with JDK 21 with the caveats that formatter is off. The non-negotiable condition is that CI still checks format. I think this is exactly what this PR is doing, whi

Re: Building with JDK 21

> > > It has a caveat (we can't run formatter on 21 and 8, and we need to choose >> one). > > > Would it format differently? I would go for 21 since that's the path > forward, but I'm also fine with JB's suggestion 👍 > >> >> Yeah it would format differently, because the underlying *google-java-form

Re: Building with JDK 21

Thanks Piotr for raising this and summing it up so far. The timing of deprecating is always hard, but it looks like there is a lot of traction within the Java data ecosystem to move to a later version: - Avro 1.12.0 will be JDK17+ - Spark 4.x will be JDK17+ - Arrow 18 will be JDK11+ -

Re: Building with JDK 21

Hi Piotr, Even if it might fail at the beginning, I think we can start Java21 on a profile/CI even if we still use Java8 by default. So, agree to: 1. Even Java21 in a profile without formatter for now), the default is still Java8 2. When Java8 is dropped and we enable Java21 by default, we also sw

Re: Building with JDK 21

Hi, Thank you all for your comments and perspectives. Summing up so far: It is clear that dropping JDK 8 is imminent, but it is also inevitably painful for some users. We don't have precise date/version when Java 8 can be dropped (along with Hive module) and Iceberg 1.7 was proposed for that. We

Re: Building with JDK 21

Hi Piotr I already commented on the PR directly, so let me share here: I'm in favor of dropping Java8 and directly jumping to Java21. However, to do this jump, I would need to remove "old" modules, like Hive. I think it's totally acceptable on a new major version. I shared this in the thread about

Re: Building with JDK 21

I'm generally in favor of dropping JDK 8 support and moving to JDK 21. The one blocker that is currently preventing us from doing that is Hive. What about deprecating the Hive stuff e.g. with 1.6.0 and dropping it with 1.7.0 and moving to the newer JDK in 1.7.0? Iceberg 1.6.0 would then be the las

Re: Building with JDK 21

I guess it depends on the generated class file version. On Wed, Jul 10, 2024 at 2:53 PM Manu Zhang wrote: > I suppose Iceberg built with Java 11/17/21 (at least one of them) can run > with Spark/Hadoop built with Java 8? > > On Wed, Jul 10, 2024 at 11:59 AM Steven Wu wrote: > >> +1 for dropping

Re: Building with JDK 21

I suppose Iceberg built with Java 11/17/21 (at least one of them) can run with Spark/Hadoop built with Java 8? On Wed, Jul 10, 2024 at 11:59 AM Steven Wu wrote: > +1 for dropping Java 8 as it is really old (3 versions behind latest Java > LTS version - 11, 17, 21). > > Here was the thread from o

Re: Building with JDK 21

+1 for dropping Java 8 as it is really old (3 versions behind latest Java LTS version - 11, 17, 21). Here was the thread from over a year ago on dropping JDK 8 support. https://lists.apache.org/thread/gq08prw7tv8q6h71lfc9bjlj074ckccv Hive's lack of Java 11 support was identified as a blocker. The

Re: Building with JDK 21

Piotr, thanks for the details. I am not sure about the community usage of Java 8. I was mostly asking for my company which is still on Java 8 as of today. On Tue, Jul 9, 2024 at 2:40 PM Piotr Findeisen wrote: > Hi > > Thanks Russel, Ryan, Pucheng for your input! > > > As an alternative I think i

Re: Building with JDK 21

Hi Thanks Russel, Ryan, Pucheng for your input! As an alternative I think it would be fine if we disable the formatter when > using Java 21 and just make sure we always have tests run with Java 8 and > the formatter checks in our CI. If we go this route I think we stay with > Java 8 for formatti

Re: Building with JDK 21

What does dropping Java 8 support mean to companies that are still using Java 8 for Iceberg in production? On Tue, Jul 9, 2024 at 9:26 AM Ryan Blue wrote: > +1 for removing Java 8 support. > > On Tue, Jul 9, 2024 at 9:24 AM Russell Spitzer > wrote: > >> The different formatting preferences soun

Re: Building with JDK 21

+1 for removing Java 8 support. On Tue, Jul 9, 2024 at 9:24 AM Russell Spitzer wrote: > The different formatting preferences sounds annoying enough that I would > think we should just drop the Java8 support. Do we have anyone who strongly > prefers keeping Java 8 support? > > As an alternative I

Re: Building with JDK 21

The different formatting preferences sounds annoying enough that I would think we should just drop the Java8 support. Do we have anyone who strongly prefers keeping Java 8 support? As an alternative I think it would be fine if we disable the formatter when using Java 21 and just make sure we alway

Building with JDK 21

Hi, Java 21 is the latest "LTS version" released GA in September 2023. Some Iceberg users already run with Java 21 on production (and FWIW Trino runs with 22 already) I thought it would be nice to add support for building and testing Iceberg with Java 21. Conceptually this is simple (see PR