Re: ROWS_DIVISOR in BaseTaskWriter

We introduced that config to avoid expensive calls to determine the current file size. It was not made configurable as we kind of expected it to work in most use cases and were reluctant to add another config nobody will ever set. I think we made a similar check configurable when writing Parquet ro

Re: Building with JDK 21

+1 for dropping JDK8 support and adding JDK21. > What does dropping Java 8 support mean to companies that are still using Java 8 for Iceberg in production? >From the AWS side, AWS Corretto JDK8 end of life is July 2026, see: https://aws.amazon.com/corretto/faqs/#support_calendar. I would suggest

Re: Building with JDK 21

+1 adding java 21 support +1 removing java 8 support On Fri, Jul 19, 2024 at 1:33 PM Daniel Weeks wrote: > I'm also in favor of removing Java 8 support. Hive docs state Hive 3 > requires java 8 and > in prior cases there were potential co

Re: [DISCUSS][BYLAWS] Moving forward on the bylaws

> specifically the discussion of the standard roles Yes, there are also other places with different definitions. For example the default project guideline [1] has additional description of the PMC member and chair responsibilities. There are a few other places like ASF glossary [2] where these ter

Re: Building with JDK 21

I'm also in favor of removing Java 8 support. Hive docs state Hive 3 requires java 8 and in prior cases there were potential correctness issues when running with newer Java versions (these may have been addressed). As long as we're not upda

ROWS_DIVISOR in BaseTaskWriter

Hello, I am writing to ask about the ROWS_DIVISOR in BaseTaskWriter. Based on this, we only roll over to a new file every 1000 rows and if the 1000 rows has reached target

Re: Building with JDK 21

I agree that if we can separate the discussion about how to support Hive, then we should do that. +1 to removing Java 8 support +1 to adding Java 21 support. On Fri, Jul 19, 2024 at 12:58 PM huaxin gao wrote: > +1 in favor of adding java 21 support > +1 in favor of removing java 8 support > > I

Re: [RESULT][VOTE] Merge table spec clarifications on time travel and equality deletes

+1, Sorry I meant to vote before. I just had nits on the wording On Fri, Jul 19, 2024 at 2:04 PM Micah Kornfield wrote: > Hi Dmitri, > Thank you for the comment, maybe we can continue the discussion on the PR > (there are still some other open issues). I don't think the current spec > reference

Re: Building with JDK 21

+1 in favor of adding java 21 support +1 in favor of removing java 8 support I am currently working on Spark 4.0 / Iceberg integration . Spark 4.0 runs on Java 17/21. On Fri, Jul 19, 2024 at 4:58 AM Piotr Findeisen wrote: > Hi, > > We recently start

Re: [VOTE] Release Apache Iceberg 1.6.0 RC1

+1 (binding) Verified sigs/sums/license/build/test (Java 17) -Dan On Fri, Jul 19, 2024 at 2:53 AM Robert Stupp wrote: > +1 (nb) > > On 18.07.24 08:37, Jean-Baptiste Onofré wrote: > > Hi everyone, > > > > I propose that we release the following RC as the official Apache > > Iceberg 1.6.0 releas

Re: [RESULT][VOTE] Merge table spec clarifications on time travel and equality deletes

Hi Dmitri, Thank you for the comment, maybe we can continue the discussion on the PR (there are still some other open issues). I don't think the current spec references the REST catalog, but I think the same issue occurs for table specification. Cheers, Micah On Fri, Jul 19, 2024 at 10:37 AM Dmi

Re: [DISCUSS][BYLAWS] Moving forward on the bylaws

Another thing that has come up is putting together a committer requirements/guidelines doc. I think it would be great to have a discussion about how we want to do that. Specifically, I'm against adding requirements for new committers (such as time-based minimums or recency requirements) that exclud

Re: [DISCUSS][BYLAWS] Moving forward on the bylaws

I meant specifically the discussion of the standard roles (eg. users, committers, pmc, pmc chair) that are well covered in https://www.apache.org/foundation/how-it-works/#roles .. Owen On Fri, Jul 19, 2024 at 10:43 AM Jack Ye wrote: > Thank you Owen for moving this forward, we heard you were si

Re: [RESULT][VOTE] Merge table spec clarifications on time travel and equality deletes

Sorry for the late reply. The vote closed, so I'll just post my comments without voting here. My reading of the spec change in PR #8982 [1] is that it is not normative. More specifically, REST catalog implementations that do not expose the full snapshot history in metadata JSON will not violate th

Re: [DISCUSS][BYLAWS] Moving forward on the bylaws

Thank you Owen for moving this forward, we heard you were sick, hope you are fully recovered now! One point regarding "referring to the Apache documentation": I am totally for that, but during the initial investigation, I found out that the Apache documentations are scattered around, and also have

Re: [DISCUSS][BYLAWS] Moving forward on the bylaws

Everyone is welcome to vote. The Iceberg PMC will have the only binding votes. .. OwenOn Jul 19, 2024, at 10:19, Wing Yew Poon wrote:Hi Owen,Thanks for doing this.Once you have the questions and choices, who gets to vote on them?- Wing YewOn Fri, Jul 19, 2024 at 10:07 AM Owen O'Malley

Re: [DISCUSS][BYLAWS] Moving forward on the bylaws

One quick followup. I'd recommend against having sub-project or specification committers. Part of being a committer is knowing what you do and don't know. I've never seen a problem at Apache where a committer committed something in an area where they had no expertise. Trying to formalize those boun

Re: [DISCUSS][BYLAWS] Moving forward on the bylaws

Hi Owen, Thanks for doing this. Once you have the questions and choices, who gets to vote on them? - Wing Yew On Fri, Jul 19, 2024 at 10:07 AM Owen O'Malley wrote: > All, >Sorry for the long pause on bylaws discussion. It was a result of > wanting to avoid the long US holiday week (July 4th

Re: [RESULT][VOTE] Merge table spec clarifications on time travel and equality deletes

The vote passes with: 5 "+1 Binding votes" 3 "+1 Non-binding votes." 0 "-1 votes" Actions to be taken: 1. Update the language/location of the clarification on time travel in https://github.com/apache/iceberg/pull/8982 and then have a committer/PMC member merge. I'll try to have this updated b

[DISCUSS][BYLAWS] Moving forward on the bylaws

All, Sorry for the long pause on bylaws discussion. It was a result of wanting to avoid the long US holiday week (July 4th) and my procrastination, which was furthered by a side conversation that asked me to consider how to move forward in an Apache way. I'd like to thank Jack for moving this

Re: [Early Feedback] Variant and Subcolumnarization Support

Agreed with point 1. For point 2, I also prefer to hold the spec and reference implementation under Iceberg. Here are the reasons: 1. It is unconventional and impractical for one engine to depend on another for data types. For instance, it is not ideal for Trino to rely on data types defined by th

Re: [VOTE] Merge table spec clarifications on time travel and equality deletes

+1 (binding) added minor comments to the time travel PR. Best, Jack Ye On Fri, Jul 19, 2024 at 8:22 AM Daniel Weeks wrote: > +1 (binding) > > Thanks, Micah. > > On Thu, Jul 18, 2024 at 8:29 PM Amogh Jahagirdar <2am...@gmail.com> wrote: > >> +1 (non-binding) on these spec clarifications >> >> T

Re: [VOTE] Merge table spec clarifications on time travel and equality deletes

+1 (binding) Thanks, Micah. On Thu, Jul 18, 2024 at 8:29 PM Amogh Jahagirdar <2am...@gmail.com> wrote: > +1 (non-binding) on these spec clarifications > > Thanks, > Amogh Jahagirdar > > On Thu, Jul 18, 2024 at 5:08 PM Steven Wu wrote: > >> I am +1 for the spec clarifications. >> >> I have left

Re: Building with JDK 21

Hi, We recently started to test Hive3 with Java 11 and 17 and the tests pass. So dropping Java 8 doesn't technically require removing the Hive 3 related modules, unless users cannot do anything useful with them (because e.g. they can only run Hive run

Re: Building with JDK 21

Back to the main topic: Removing java8 support still allows us to use the old HMS libraries, so iceberg-hive-metastore module is not affected in this sense. We still need to run the tests, to ensure that there is no changes in the behaviour, but it is unlikely. +1 in favor of adding java 21 suppo

Re: [VOTE] Release Apache Iceberg 1.6.0 RC1

+1 (nb) On 18.07.24 08:37, Jean-Baptiste Onofré wrote: Hi everyone, I propose that we release the following RC as the official Apache Iceberg 1.6.0 release. The commit ID is 229d8f6fcd109e6c8943ea7cbb41dab746c6d0ed * This corresponds to the tag: apache-iceberg-1.6.0-rc1 * https://github.com/ap