Re: Dropping JDK 8 support

Hey everyone, I'm also in favor of dropping JDK8. To give some context in the ecosystem, next to Spark 4, a lot of projects are moving beyond Java 8: 1. Arrow dropped JDK8 support last week which will be part of the next 18.0.0 release. 2.

Re: Dropping JDK 8 support

I apologize. I didn't realize this was a formal voting thread. I thought it was for discussion and voicing concerns. On Tue, Jul 23, 2024 at 10:56 PM Wing Yew Poon wrote: > I just wish to point out that when people started voting, the proposal was > "dropping JDK 8 support in Iceberg 2.0 release

Re: Dropping JDK 8 support

+1 (nb) in general, and +1 (nb) on dropping Java 8 in Iceberg 1.7 On 23.07.24 18:54, Steve Zhang wrote: +1 (non-binding) Thanks, Steve Zhang On Jul 22, 2024, at 10:13 PM, Ajantha Bhat wrote: +1 (non-binding) -- Robert Stupp @snazy

Re: [ANNOUNCE] Welcoming new committers and PMC members

Thank you all! Congratulations to Kevin, Piotr, Sung, Xuanwo, and Renjie! It’s truly an honor to contribute to the project and witness the incredible growth of our community. I’m excited to continue working together and help drive the project to new heights! On Tue, Jul 23, 2024 at 7:24 PM Renjie

Re: Dropping JDK 8 support

I just wish to point out that when people started voting, the proposal was "dropping JDK 8 support in Iceberg 2.0 release". It's fine for people to propose dropping JDK8 support sooner than that (and I'm not against that), but the proposal being voted on should not be switched mid-vote. - Wing Yew

Re: Dropping JDK 8 support

I understand that transitioning from JDK 8 to JDK 11 requires some effort from the users. However, even if we wait until version 2.0, we still encounter the same problem. I don't see the need for more time to test the discontinuation of JDK 8 support. The configuration of Spark 3.5 with JDK 11 and

Re: Dropping JDK 8 support

Yes, I'm asking for users who use JDK 8 with Spark 3.5. Users can continue to use 1.6+ with Spark 3.5 and JDK 8 if we continue to support them. If we drop JDK 8 support after 1.6, then there might be issues for Spark 3.5 with JDK 8 users. I'm +1 to drop JDK 8 support in 2.0. I think it's worth mor

Re: Dropping JDK 8 support

Hi Manu, Thanks for the discussion. Is your concern about customers who use JDK 8 with Spark 3.5? But we will face the same problem if we drop JDK 8 in Iceberg 2.0, unless we plan to drop Spark 3.5 support in 2.0. Huaxin On Tue, Jul 23, 2024 at 7:30 PM Renjie Liu wrote: > Hi, Manu: > > > If we

Re: Dropping JDK 8 support

Hi, Manu: > If we drop JDK 8 support in 1.7, can Iceberg 1.7+ work seamlessly with Spark 3.5? Otherwise, users might get stuck in 1.6. I think spark 3.5 supports JDK 8/11/17 according to their doc. So users could still use iceberg 1.7+ after upgrading JDK. On Wed, Jul 24, 2024 at 9:40 AM Manu Zh

Re: [ANNOUNCE] Welcoming new committers and PMC members

Thanks everyone! And congratulations to Pitor, Kevin, Sung, Xuanwo, Honah! It's awesome to work within and see the growth of this community! On Wed, Jul 24, 2024 at 9:41 AM Manu Zhang wrote: > Congrats everyone! Well done! > > Regards, > Manu > > On Wed, Jul 24, 2024 at 4:48 AM Piotr Findeisen

Re: [ANNOUNCE] Welcoming new committers and PMC members

Congrats everyone! Well done! Regards, Manu On Wed, Jul 24, 2024 at 4:48 AM Piotr Findeisen wrote: > Dear all, > thank you for your trust. Very much appreciated > > Kevin, Sung, Xuanwo, Honah, Renjie -- congratulations! it's awesome that > your efforts were noticed and the value you bring to th

Re: Dropping JDK 8 support

Not sure about other engines but Spark has JDK 8 support till 3.5, which looks like a LTS version. If we drop JDK 8 support in 1.7, can Iceberg 1.7+ work seamlessly with Spark 3.5? Otherwise, users might get stuck in 1.6. >

Re: [DISCUSS] Deprecate HadoopTableOperations, move to tests in 2.0

Sir, regarding this point, we have some experience. In my view, as long as the file system supports atomic single-file writing, where the file becomes immediately visible upon the client's successful write operation, that is sufficient. We can do without the rename operation as long as the fil

Re: [ANNOUNCE] Welcoming new committers and PMC members

Dear all, thank you for your trust. Very much appreciated Kevin, Sung, Xuanwo, Honah, Renjie -- congratulations! it's awesome that your efforts were noticed and the value you bring to the table -- recognized. Best, Piotr On Tue, 23 Jul 2024 at 18:56, Steve Zhang wrote: > Congrats everyone! >

Re: Re: [DISCUSS] Deprecate HadoopTableOperations, move to tests in 2.0

If we come up with a new storage-only catalog implementation that could solve those limitations and also leverage the new features being developed in object storage, would that be a potential alternative strategy? so HadoopCatalog users has a way to move forward with still a storage-only catalog th

Re: [Early Feedback] Variant and Subcolumnarization Support

Thanks folks for additional discussion. There are some questions related to subcolumniziation (spark shredding - see the discussion ) and we would like to host another meeting to mainly discuss that since we plan to adopt it. We can also follow up the Sp

Re: Dropping JDK 8 support

Thanks Ryan for the reply! +1 in dropping JDK 8 support in 1.7. On Tue, Jul 23, 2024 at 9:59 AM Ryan Blue wrote: > I don't have an issue with dropping JDK 8 support in 1.7. I think it is > better not to align changes like this with major releases because it makes > major releases harder because

Re: Re: [DISCUSS] Deprecate HadoopTableOperations, move to tests in 2.0

I don't think we would want to put this in a module with other catalog implementations. It has serious limitations and is actively discouraged, while the other catalog implementations still have value as either REST back-end catalogs or as regular catalogs for many users. On Tue, Jul 23, 2024 at 9

Re: Dropping JDK 8 support

I don't have an issue with dropping JDK 8 support in 1.7. I think it is better not to align changes like this with major releases because it makes major releases harder because we're trying to get more things in a release. Putting out a major release just for breaking API changes makes the most sen

Re: [ANNOUNCE] Welcoming new committers and PMC members

Congrats everyone! Thanks, Steve Zhang > On Jul 23, 2024, at 9:20 AM, Anton Okolnychyi wrote: > > Congrats everyone!

Re: Dropping JDK 8 support

+1 (non-binding) Thanks, Steve Zhang > On Jul 22, 2024, at 10:13 PM, Ajantha Bhat wrote: > > +1 (non-binding)

Re: Dropping JDK 8 support

+1 On Tue, Jul 23, 2024 at 11:47 AM huaxin gao wrote: > We don't have to drop JDK11 support. In spark-ci, I can change the matrix > to only run Java 17 for Spark 4.0, but in java-ci, we might not be able to > build java docs and do build checks for JDK 11. > > Huaxin > > > > On Tue, Jul 23, 2024

Re: [ANNOUNCE] Welcoming new committers and PMC members

Congrats everyone! Thanks, Steve Zhang > On Jul 23, 2024, at 9:20 AM, Anton Okolnychyi wrote: > > Congrats everyone!

Re: [ANNOUNCE] Welcoming new committers and PMC members

Congrats everyone! Thanks for all the great work and looking forward to more. -Dan On Tue, Jul 23, 2024 at 9:20 AM Anton Okolnychyi wrote: > Congrats everyone! > > вт, 23 лип. 2024 р. о 09:11 Yufei Gu пише: > >> Congratulations! Thanks a lot for the contribution! >> >> Yufei >> >> >> On Tue,

Re: Dropping JDK 8 support

We don't have to drop JDK11 support. In spark-ci, I can change the matrix to only run Java 17 for Spark 4.0, but in java-ci, we might not be able to build java docs and do build checks for JDK 11. Huaxin On Tue, Jul 23, 2024 at 9:32 AM Jack Ye wrote: > Does that mean we also need to drop JDK1

Re: Dropping JDK 8 support

Does that mean we also need to drop JDK11 support? I think there should be a way to configure CI to only run JDK17 for Spark 4.0: https://github.com/apache/iceberg/blob/main/.github/workflows/spark-ci.yml#L74 https://stackoverflow.com/questions/68994484/how-to-skip-a-configuration-of-a-matrix-wit

Re: [ANNOUNCE] Welcoming new committers and PMC members

Congrats everyone! вт, 23 лип. 2024 р. о 09:11 Yufei Gu пише: > Congratulations! Thanks a lot for the contribution! > > Yufei > > > On Tue, Jul 23, 2024 at 9:01 AM Walaa Eldin Moustafa < > wa.moust...@gmail.com> wrote: > >> Congratulations everyone! Great to see the community growing. >> >> Than

Re: Dropping JDK 8 support

It seems my earlier question might have been overlooked. Could we clarify if JDK 8 support is being dropped in the next version? The proposal indicated for Iceberg 2.0 release, but it's unclear if that's our next version. Given that I'm working on Spark 4.0 support

Re: Re: [DISCUSS] Deprecate HadoopTableOperations, move to tests in 2.0

For some additional information, we also have some Iceberg HDFS users on EMR. Those are mainly users that have long-running Hadoop and HBase installations. They typically refresh their installation every 1-2 years. >From my understanding, they use S3 for data storage, but metadata is kept in the lo

Re: [ANNOUNCE] Welcoming new committers and PMC members

Congratulations! Thanks a lot for the contribution! Yufei On Tue, Jul 23, 2024 at 9:01 AM Walaa Eldin Moustafa wrote: > Congratulations everyone! Great to see the community growing. > > Thanks, > Walaa. > > On Tue, Jul 23, 2024 at 8:51 AM Alex Dutra > wrote: > >> Congratulations to you all! >

Re: [ANNOUNCE] Welcoming new committers and PMC members

Thank you, Fokko! And thank you to everyone for contributing to a great open-source community. It’s been a pleasure to be part of this project. Looking forward to the future! On Tue, Jul 23, 2024 at 8:59 AM Walaa Eldin Moustafa wrote: > Congratulations everyone! Great to see the community growi

Re: [ANNOUNCE] Welcoming new committers and PMC members

Congrats all, and thank you for your contributions! Thanks, Amogh Jahagirdar On Tue, Jul 23, 2024 at 9:59 AM Walaa Eldin Moustafa wrote: > Congratulations everyone! Great to see the community growing. > > Thanks, > Walaa. > > On Tue, Jul 23, 2024 at 8:51 AM Alex Dutra > wrote: > >> Congratula

Re: [ANNOUNCE] Welcoming new committers and PMC members

Congratulations everyone! Great to see the community growing. Thanks, Walaa. On Tue, Jul 23, 2024 at 8:51 AM Alex Dutra wrote: > Congratulations to you all! > > Thanks, > Alex > > On Tue, Jul 23, 2024 at 5:30 PM Jack Ye wrote: > >> Congratulations!! >> >> Best, >> Jack Ye >> >> On Tue, Jul 23,

Re: [ANNOUNCE] Welcoming new committers and PMC members

Congratulations to you all! Thanks, Alex On Tue, Jul 23, 2024 at 5:30 PM Jack Ye wrote: > Congratulations!! > > Best, > Jack Ye > > On Tue, Jul 23, 2024 at 8:16 AM Ryan Blue > wrote: > >> Congratulations, everyone! And thanks for all your contributions! >> >> On Tue, Jul 23, 2024 at 8:06 AM Me

Re: Re: [DISCUSS] Deprecate HadoopTableOperations, move to tests in 2.0

Thanks for the context, lisoda. I agree that it's good to understand the issues you're facing with the HadoopCatalog. One follow up question that I have is what the underlying storage is. Are you using HDFS for those 30,000 customers? I think you're right that there is a challenge to migrating. Be

Re: [DISCUSS][BYLAWS] Moving forward on the bylaws

+1 on the list Micah provides, and ideally most of the matters in 3 and 5 could directly be deferred to ASF guidelines. -Jack On Tue, Jul 23, 2024 at 8:30 AM Ryan Blue wrote: > I agree with Micah as well. I also don't want to get bogged down in > bureaucracy and more rules inevitably lead to mo

Re: [ANNOUNCE] Welcoming new committers and PMC members

Congratulations!! Best, Jack Ye On Tue, Jul 23, 2024 at 8:16 AM Ryan Blue wrote: > Congratulations, everyone! And thanks for all your contributions! > > On Tue, Jul 23, 2024 at 8:06 AM Mehul Batra > wrote: > >> Congratulations Everyone! >> >> On Tue, Jul 23, 2024 at 6:36 PM Fokko Driesprong >

Re: [DISCUSS][BYLAWS] Moving forward on the bylaws

I agree with Micah as well. I also don't want to get bogged down in bureaucracy and more rules inevitably lead to more bureaucracy. I also like the prioritization he suggests. I also think it's important to clarify that this thread isn't trying to rehash the previous threads in a new place. I thin

Re: Dropping JDK 8 support

+1 On Mon, Jul 22, 2024 at 10:49 PM Péter Váry wrote: > +1 (non-binding) > > On Tue, Jul 23, 2024, 07:15 Ajantha Bhat wrote: > >> +1 (non-binding) >> >> On Tue, Jul 23, 2024 at 9:54 AM Yufei Gu wrote: >> >>> Hi Manu, >>> >>> If JDK 8 support is dropped in 2.0, will we continue to fix critical

Re: [ANNOUNCE] Welcoming new committers and PMC members

Congratulations, everyone! And thanks for all your contributions! On Tue, Jul 23, 2024 at 8:06 AM Mehul Batra wrote: > Congratulations Everyone! > > On Tue, Jul 23, 2024 at 6:36 PM Fokko Driesprong wrote: > >> Hi everyone, >> >> The Iceberg PMC is excited to announce new committers and PMC memb

Re: [ANNOUNCE] Welcoming new committers and PMC members

Congratulations Everyone! On Tue, Jul 23, 2024 at 6:36 PM Fokko Driesprong wrote: > Hi everyone, > > The Iceberg PMC is excited to announce new committers and PMC members to > the Apache Iceberg project. > > New committers: > > >- > >Kevin Liu (kevinjqliu) >- > >Piotr Findeisen (

Re: [DISCUSS][BYLAWS] Moving forward on the bylaws

Micah has a great list there for me. I'm similarly not as interested in the bureaucracy of the project and more interested in actually discussing how we operate from a technical perspective as the community grows. On Tue, Jul 23, 2024 at 1:01 AM Micah Kornfield wrote: > My 2 cents on this topic.

Re: [ANNOUNCE] Welcoming new committers and PMC members

Congrats everybody! :) On Tue, Jul 23, 2024 at 10:23 AM Gang Wu wrote: > Congrats! > > On Tue, Jul 23, 2024 at 10:17 PM Russell Spitzer < > russell.spit...@gmail.com> wrote: > >> "so many" :) >> >> On Tue, Jul 23, 2024 at 9:14 AM Russell Spitzer < >> russell.spit...@gmail.com> wrote: >> >>> This

Re: [ANNOUNCE] Welcoming new committers and PMC members

Congratulations everyone! On Tue, Jul 23, 2024 at 7:52 PM Gang Wu wrote: > Congrats! > > On Tue, Jul 23, 2024 at 10:17 PM Russell Spitzer < > russell.spit...@gmail.com> wrote: > >> "so many" :) >> >> On Tue, Jul 23, 2024 at 9:14 AM Russell Spitzer < >> russell.spit...@gmail.com> wrote: >> >>> Th

Re: [ANNOUNCE] Welcoming new committers and PMC members

Congrats! On Tue, Jul 23, 2024 at 10:17 PM Russell Spitzer wrote: > "so many" :) > > On Tue, Jul 23, 2024 at 9:14 AM Russell Spitzer > wrote: > >> This is truly an exciting day. To have to many qualified folks being >> recognized by the Iceberg project fills me with pride. I can't wait to see >

Re: [ANNOUNCE] Welcoming new committers and PMC members

"so many" :) On Tue, Jul 23, 2024 at 9:14 AM Russell Spitzer wrote: > This is truly an exciting day. To have to many qualified folks being > recognized by the Iceberg project fills me with pride. I can't wait to see > what we get done together! > > On Tue, Jul 23, 2024 at 9:12 AM Sung Yun wrote

Re: [ANNOUNCE] Welcoming new committers and PMC members

This is truly an exciting day. To have to many qualified folks being recognized by the Iceberg project fills me with pride. I can't wait to see what we get done together! On Tue, Jul 23, 2024 at 9:12 AM Sung Yun wrote: > Thank you very much! > > I am excited to see the project growing to new cap

Re: [ANNOUNCE] Welcoming new committers and PMC members

Congrats everyone! On Tuesday, July 23, 2024, Sung Yun wrote: > Thank you very much! > > I am excited to see the project growing to new capacities as well, and to > be an active part of that journey. > > I will continue to work hard together with the community to take > (Py)Iceberg to its next s

Re: [ANNOUNCE] Welcoming new committers and PMC members

Thank you very much!I am excited to see the project growing to new capacities as well, and to be an active part of that journey.I will continue to work hard together with the community to take (Py)Iceberg to its next steps.SungOn Jul 23, 2024, at 10:01 AM, Matt Topol wrote:Congrats!! On Tue, Jul

Re: [ANNOUNCE] Welcoming new committers and PMC members

Congrats!! On Tue, Jul 23, 2024, 9:52 AM Xuanwo wrote: > Thank you so much for the recognition! I'm thrilled to join as an Iceberg > Committer. > > I'm currently working hard on the iceberg rust project. If you're > interested in rust, feel free to join us! > > On Tue, Jul 23, 2024, at 21:03, Fo

Re: [ANNOUNCE] Welcoming new committers and PMC members

Thank you so much for the recognition! I'm thrilled to join as an Iceberg Committer. I'm currently working hard on the iceberg rust project. If you're interested in rust, feel free to join us! On Tue, Jul 23, 2024, at 21:03, Fokko Driesprong wrote: > Hi everyone, > > > The Iceberg PMC is exc

Re: [Early Feedback] Variant and Subcolumnarization Support

I'm late replying to this but I'm also in agreement with 1 (adopting the spark variant encoding), 3 (specifically only having a variant type), and 4 (ensuring we are thinking through subcolumnarization upfront since without it the variant type may not be that useful). I'd also support having the s

[ANNOUNCE] Welcoming new committers and PMC members

Hi everyone, The Iceberg PMC is excited to announce new committers and PMC members to the Apache Iceberg project. New committers: - Kevin Liu (kevinjqliu) - Piotr Findeisen (findepi) - Sung Yun (syun64) - Xuanwo (xuanwo) New members of the PMC: - Honah (ho