Re: [VOTE] Release Apache Iceberg 1.9.1 RC1

+1 (non-binding). Verified against Snowflake engine. On Tue, May 27, 2025 at 9:28 AM huaxin gao wrote: > +1 (non-binding) > Verified signature, checksum, license and ran some tests. > > On Tue, May 27, 2025 at 9:06 AM Russell Spitzer > wrote: > >> For all those who haven't seen this before, GP

Re: [VOTE] Release Apache Iceberg 1.9.1 RC0

from newer clients? >> >>> >> >>> So in this case, newer clients are trying to express deletions with >> larger sizes and the server is unable to handle it due to the assertion in >> the older implementation, not because the protocol changed. Though I c

Re: [VOTE] Adopt the v3 spec changes

+1 (non-binding). Great to see we are moving forward. On Mon, May 19, 2025 at 9:06 PM Denny Lee wrote: > +1 (non-binding) > > On Mon, May 19, 2025 at 9:04 PM Yufei Gu wrote: > >> +1 (binding) >> >> Yufei >> >> >> On Mon, May 19, 2025 at 8:59 PM Amogh Jahagirdar <2am...@gmail.com> >> wrote: >>

Re: [VOTE] Release Apache Iceberg 1.9.1 RC0

d and can get the benefits of the > change in the 1.9 release. > > If we think it's simpler to just revert for 1.9 and cycle these proposed > server changes for 1.10 and then the client changes for the release after > 1.10, I think I'm OK (+0) with that as well. > > Than

Re: Spark 4.0/Iceberg Integration Merged – Spark 3.5 Merges Can Resume

Thanks a lot to get that merged! Huaxin. On Fri, May 16, 2025 at 12:05 PM Amogh Jahagirdar <2am...@gmail.com> wrote: > Thanks Huaxin for your patience while getting this through! > > On Fri, May 16, 2025 at 12:20 PM Steve wrote: > >> Thanks Huaxin! Great work and looking forward to the spark 4.0

Re: [VOTE] Release Apache Iceberg 1.9.1 RC0

assertion in >>> the older implementation, not because the protocol changed. Though I can >>> see the grey area in that it either forces clients to not upgrade for Java >>> server implementations which haven't upgraded OR it server implementations >>> end up

Re: [VOTE] Release Apache Iceberg 1.9.1 RC0

I have verified RC against Snowflake build. Everything works except one issue introduced by https://github.com/apache/iceberg/pull/12670/ : the client with 1.9.x can't work with the catalog server with old library to remove the snapshots since the the client now will remove the snapshots in bulk w

Re: [Discuss] Iceberg 1.9.1 Release

Is https://github.com/apache/iceberg/pull/12670/ considered introducing a broken change for catalog in 1.9.0? Now the client can send a list of snapshots to the server to remove in TableMetadata class while the server with the old Iceberg library still checks to make sure only a single snapshotId i

Re: [Discuss] spec: Add missing 'spec_id' for data_file

Szehon pointed out that it's not serialized to the manifest file. So it's used internally but it doesn't need to be in the spec. Thanks, Aihua On Mon, May 5, 2025 at 10:34 AM Aihua Xu wrote: > Hi all, > > I notice that we are missing spec_id for data_file in the sp

[Discuss] spec: Add missing 'spec_id' for data_file

Hi all, I notice that we are missing spec_id for data_file in the spec while working on V3 features (https://github.com/apache/iceberg/pull/12970) and it seems obvious. But let me know if a vote is needed for such change. Thanks, Aihua

Re: [VOTE] Add encryption keys to table metadata

+1 (non-binding). On Wed, Apr 30, 2025 at 3:49 PM Daniel Weeks wrote: > +1 (binding) > > On Wed, Apr 30, 2025 at 1:38 PM Amogh Jahagirdar <2am...@gmail.com> wrote: > >> +1 (binding) >> >> On Wed, Apr 30, 2025 at 1:29 PM Anurag Mantripragada >> wrote: >> >>> +1 (non-binding) >>> >>> ~ Anurag Man

Re: [VOTE] Release Apache Iceberg 1.9.0 RC2

+1 (non-binding) 1. Run the tests 2. Test against Snowflake build Thanks Aihua On Sat, Apr 26, 2025 at 7:02 AM Amogh Jahagirdar <2am...@gmail.com> wrote: > +1 (binding) > > 1. Verified sigs/checksum/license > 2. Ran build/test with JDK 17 > > Thanks Ajantha for running the release! > > On Fri,

Re: [VOTE] Spec Update: Variant Field Lower/Upper Bounds

This passes with 13 votes (+6 bindings and +7 non-bindings) and no -1 or +0 votes. Thanks everyone! On Mon, Apr 21, 2025 at 1:26 PM Aihua Xu wrote: > +1 (non-binding) from me. > > On Sun, Apr 20, 2025 at 12:19 PM Denny Lee wrote: > >> +1 (non-binding) >> >> On Su

Re: [VOTE] Spec Update: Variant Field Lower/Upper Bounds

+1 (non-binding) from me. On Sun, Apr 20, 2025 at 12:19 PM Denny Lee wrote: > +1 (non-binding) > > On Sun, Apr 20, 2025 at 11:22 Fokko Driesprong wrote: > >> +1 (binding) >> >> Op zo 20 apr 2025 om 05:59 schreef Daniel Weeks : >> >>> +1 (binding) >>> >>> On Sat, Apr 19, 2025, 12:07 AM Gang Wu

[VOTE] Spec Update: Variant Field Lower/Upper Bounds

Hi all, I'd like to initiate a vote to include a spec update for supporting lower and upper bounds on Variant fields. Summary of the change: The writer determines which fields to collect bounds for a Variant column. Field bounds are stored as serialized Variant objects, where each key is a normal

Re: [VOTE] Update row lineage spec ID assignment

+ (non-binding). On Thu, Apr 17, 2025 at 11:22 AM Steven Wu wrote: > +1 (binding) > > On Thu, Apr 17, 2025 at 11:09 AM Amogh Jahagirdar <2am...@gmail.com> > wrote: > >> +1 (binding) >> >> On Thu, Apr 17, 2025 at 11:54 AM Szehon Ho >> wrote: >> >>> +1 (binding) Seems cleaner to me. >>> >>> Than

Re: [VOTE] Release Apache Iceberg 1.9.0 RC0

+1 (non-binding). Verified against Snowflake build. On Sun, Apr 13, 2025 at 12:20 AM Yuya Ebihara < yuya.ebih...@starburstdata.com> wrote: > +1 (non-binding) > > Trino Iceberg tests with 1.9.0 RC0 all passed except for the V3 write > test, which failed as expected since row lineage is now requir

Variant bound spec

Hi everyone, I'm updating the spec to define the lower/upper bounds for shredded Variant PR#12658 . Basically we are using a Variant object to hold the bound value for each shredded subcolumn as discussed in the community sync. Let me know if you have

Re: [VOTE] Release Apache Iceberg 1.8.1 RC1

ding) from me as well. > > Thanks, > > Alex > > On Thu, Feb 27, 2025 at 6:45 PM Aihua Xu wrote: > >> Hi Alex, >> >> I checked our code further. We have an internal implementation >> for RESTClient which needs an update after your change to place the token

Re: [VOTE] Release Apache Iceberg 1.8.1 RC1

change to get the token from the headers of DefaultAuthSession. With that, +1 (non-binding) Thanks, Aihua On Thu, Feb 27, 2025 at 9:26 AM Alex Dutra wrote: > Hi Aihua Xu, > > I reviewed your PR but without further details I do not agree with your > change, and I am unable to reprodu

Re: [VOTE] Release Apache Iceberg 1.8.1 RC1

gt; the OAuth2 server URI configuration and defaults to > http://127.0.0.1:8181/api/catalog/v1/oauth/tokens. This automatic > fallback will be removed in a future Iceberg release.It is recommended to > configure the OAuth2 endpoint using the 'oauth2-server-uri' property to be > prepa

Re: [VOTE] Release Apache Iceberg 1.8.1 RC1

I tested 1.8.1 RC with Snowflake build. I'm seeing the following (I'm not seeing that in 1.7.x). "exception": "java.io.IOException: *Authorization header is missing*\n\tat org.apache.polaris.service.dropwizard.auth.PolarisPrincipalAuthenticator.filter(PolarisPrincipalAuthenticator.java:43)\n\tat o

Re: [VOTE] Allow Row-Lineage with Equality Deletes

+1 (non-binding). On Thu, Feb 20, 2025 at 9:41 AM Huang-Hsiang Cheng wrote: > +1 (non-binding) > > Thanks, > Huang-Hsiang > > On Feb 20, 2025, at 9:37 AM, huaxin gao wrote: > > +1 (non-binding) > > Thanks Russell! > > On Thu, Feb 20, 2025 at 1:57 AM Fokko Driesprong wrote: > >> +1 >> >> Thanks

Re: [VOTE] Add Geometry and Geography types for V3

+1 (non-binding). On Fri, Feb 7, 2025 at 8:12 AM Jean-Baptiste Onofré wrote: > +1 > > That's a great progress ! Thanks ! > > Regards > JB > > On Thu, Feb 6, 2025 at 9:01 PM Szehon Ho wrote: > > > > Hi everyone > > > > We would like to add Geometry and Geography types to the Iceberg V3 spec: > >

Re: Welcome Huaxin Gao as a committer!

Congrats Huaxin!. On Fri, Feb 7, 2025 at 1:35 AM Manish Malhotra < manish.malhotra.w...@gmail.com> wrote: > Congratulations Huaxin! > > On Thu, Feb 6, 2025 at 6:34 PM Ron Liu wrote: > >> Congrats >> >> Best, >> Ron >> >> roryqi 于2025年2月7日周五 10:21写道： >> >>> Congrats! >>> >>> Anton Okolnychyi 于2

Re: guideline for interface change

t; Fokko > > Op zo 2 feb 2025 om 21:12 schreef Aihua Xu : > >> Thanks folks for suggestions. I will keep the existing one and mark it as >> deprecated just in case it's getting used elsewhere. I will add a new one. >> >> On Sat, Feb 1, 2025 at 2:40 AM Russell

Re: guideline for interface change

>> the new method using the old one? >> >> This would keep the old functionality until the deprecated method is >> removed. >> >> On Sat, Feb 1, 2025, 02:01 Aihua Xu wrote: >> >>> Hi folks, >>> >>> What is the general guideline

guideline for interface change

Hi folks, What is the general guideline for interface change? I'm trying to change PrimitiveType Types::fromPrimitiveString() => Type Types::fromTypeString() in https://github.com/apache/iceberg/pull/11831/files#diff-736caed551a388d34b08f223954ae7ecb2fdac9d90a4098ceedd95207d7efd4dR1149-R1152 to

Re: [VOTE] Add Variant type to Iceberg Spec

Thanks Russell for following up on this. On Wed, Jan 29, 2025 at 2:50 PM Russell Spitzer wrote: > Given that I think we are all in agreement, and Micah has signed off on > the PR I think we can go ahead and merge this one. > > On Tue, Dec 10, 2024 at 12:16 AM Aihua Xu wrote: > &

Re: [DISCUSS/VOTE] Add in ChangeLog Reserved Field IDs to Spec and Decrement Row Lineage Reserved IDs

+1 (non-binding). Thanks for fixing it. On Sun, Jan 26, 2025 at 11:30 AM Anton Okolnychyi wrote: > +1 good catch > > нд, 26 січ. 2025 р. о 08:33 Steve Zhang > пише: > >> +1 (non-binding) >> >> Thanks, >> Steve Zhang >> >> >> >> On Jan 25, 2025, at 10:48 AM, huaxin gao wrote: >> >> +1 (non-bin

Re: [DISCUSS, VOTE] OpenAPI Metadata Update for EnableRowLineage

+ (non binding). Thanks Russell. On Thu, Jan 23, 2025 at 2:05 AM Jean-Baptiste Onofré wrote: > +1 (non binding) > > Regards > JB > > On Wed, Jan 22, 2025 at 11:51 PM Russell Spitzer > wrote: > > > > Hey Y'all > > > > Yet another Row Lineage Spec update. This adds a MetadataUpdate > EnableRowLi

Re: [VOTE] Document Snapshot Summary Optional Fields as Subsection of Appendix F in Spec

+1 non-binding. On Wed, Jan 22, 2025 at 7:57 AM Kevin Liu wrote: > +1 non-binding > > Third time's the charm! :) > > On Wed, Jan 22, 2025 at 4:44 AM Jean-Baptiste Onofré > wrote: > >> +1 (non binding) >> >> Regards >> JB >> >> On Tue, Jan 21, 2025 at 9:19 PM Honah J. wrote: >> > >> > Hi everyo

[Proposal] Iceberg Metadata for Variant Shredding

Hi all, I'm trying to propose the metadata change for variant shredding in ( https://docs.google.com/document/d/1gAvt0x_ez89O8y-YqkCdMnTEykb-583YslYOgzf5sPg/edit?tab=t.0) and https://github.com/apache/iceberg/issues/11958. Basically, in Iceberg format V3, we collect subcolumn lower/upper bounds i

Re: [VOTE] Add Variant type to Iceberg Spec

that scenario >>> is unlikely) >>> 3. There is very little in our change set here that specifically >>> references the Parquet spec except for our reference link to it. >>> >>> I don't think there is anything that will happen in the spec that

Re: [DISCUSS] Apache Iceberg Summit 2025 - Selection Committee

Hi JB, I would love to help. Thanks, Aihua On Tue, Dec 3, 2024 at 4:37 AM Anurag Mantripragada wrote: > Thanks for the email. I’m happy to help as well. > > > Anurag Mantripragada > > On Dec 1, 2024, at 7:27 PM, Nick Riasanovsky wrote: > > Happy to volunteer as well. > > - Nick Riasanovsky >

[VOTE] Add Variant type to Iceberg Spec

Hi everyone, I've updated the Iceberg spec to include the new Variant type as part of #10831 . The changes are basically complete. This is a heads-up about the upcoming change. Please review and +1 to acknowledge, so we will merge. Thanks, Aihua

Re: [VOTE] Deprecate and remove last-column-id

+1 non-binding. Thanks for driving this. On Tue, Nov 19, 2024 at 5:50 PM Renjie Liu wrote: > +1, thanks Fokko! > > On Wed, Nov 20, 2024 at 8:45 AM Steve Zhang > wrote: > >> +1 nb >> >> Thanks, >> Steve Zhang >> >> >> >> On Nov 19, 2024, at 12:18 AM, Fokko Driesprong wrote: >> >> Hi everyone,

Re: [DISCUSS] Iceberg 1.7.1 release

That makes sense. Originally I thought wasb scheme change was a feature change. I will wait for 1.8.0. then. On Fri, Nov 15, 2024 at 7:18 AM Bryan Keller wrote: > I agree, we should leave this one for 1.8. > -Bryan > > On Nov 14, 2024, at 9:22 PM, Aihua Xu > wrote: > >

Re: [DISCUSS] Iceberg 1.7.1 release

Hi Bryan, I would like to include the following in 1.7.1 if possible. I keep the change minimum and I will try to get this merged soon. Add Variant data type to API: https://github.com/apache/iceberg/pull/11324 Thanks, Aihua On Thu, Nov 14, 2024 at 11:21 AM Bryan Keller wrote: > Hi all, > > A

Re: [VOTE] Release Apache Iceberg 1.7.0 RC1

+1 (non-binding) Verified signature, checksums, license Check on Spark with some manual testing Run some unit tests and integration tests on Snowflake Thanks Russell to drive this. Aihua

Bayarea Iceberg meetup in November

Hi community! The Apache Iceberg community is gathering in San Francisco on November 4th. Whether you’re interested in presenting or just want to join us for some networking, you can find the event details and RSVP here: https://lu.ma/fholq6oz . Hope to see you there! Thanks, Aihua

Re: [DISCUSS] Variant Spec Location

Le Dem wrote: > Thank you Gang, that's sounds like a good idea to me as well > > On Fri, Aug 23, 2024 at 8:47 AM Aihua Xu > wrote: > >> Thanks Gang for initiating the discussion. >> >> On Fri, Aug 23, 2024 at 2:22 AM Gang Wu wrote: >> >>

Re: [DISCUSS] Variant Spec Location

Thanks Gang for initiating the discussion. On Fri, Aug 23, 2024 at 2:22 AM Gang Wu wrote: > Thanks Aihua! > > I've started the discussion in dev@parquet: > https://lists.apache.org/thread/6h58hj39lhqtcyd2hlsyvqm4lzdh4b9z > > Best, > Gang > > On Fri, Aug 23,

Re: [DISCUSS] Variant Spec Location

gt; collaborate >> >>>> in >> >>>>>>> doing that. One thing that I don't want to do is have the Iceberg >> >>>>> project >> >>>>>>> stay in a holding pattern without any clear road

Re: [DISCUSS] Variant Spec Location

Thanks Russell for bringing this up. This is the main blocker to move forward with the Variant support in Iceberg and hopefully we can have a consensus. To me, I also feel it makes more sense to move the spec into Iceberg rather than Spark engine owns it and we try to keep it compatible with Spark

Bayarea Apache Iceberg meeting - Sept. 2024

Hi community, We're thrilled to announce an upcoming Apache Iceberg Community Meetup in the Bay Area! This is a fantastic opportunity to connect with fellow enthusiasts, share insights, and dive into the latest developments in the Apache Iceberg ecosystem. 📅 Date: September 5th, 2024 ⏰ Time: 5:0

Re: [Early Feedback] Variant and Subcolumnarization Support

general, and I agree we should avoid >> significantly complicating the design for an uncommon use case. I would >> like to understand the implication for operations like file compaction. Is >> it now up to this engine's discretion for how to combine files and if >> deci

Re: [Early Feedback] Variant and Subcolumnarization Support

f not modifying >> decimal scales, but I'm curious if this distinction satisfies everyone. If >> we remove the wording from the proposal that recommends modifying decimals >> and leave this to the engine, do we have agreement? >> >> On Thu, Jul 25, 2024 at 6:46 PM Aihua X

Re: [Early Feedback] Variant and Subcolumnarization Support

at 10:14 AM Aihua Xu wrote: > Yes. This time I was able to record it and I will share it when it’s > processed. > > > On Jul 25, 2024, at 10:01 AM, Amogh Jahagirdar <2am...@gmail.com> wrote: > >  > Any chance this meeting was recorded? I couldn't make it

Re: [Early Feedback] Variant and Subcolumnarization Support

irdarOn Tue, Jul 23, 2024 at 11:30 AM Aihua Xu <aihu...@gmail.com> wrote:Thanks folks for additional discussion.There are some questions related to subcolumniziation (spark shredding - see the discussion) and we would like to host another meeting to mainly discuss that since we plan to adop

Re: [Early Feedback] Variant and Subcolumnarization Support

>>> a superset of the Iceberg type system (UUID is supported as a canonical >>> extension type [2]). >>> >>> For point 4 subcolumnarization, I think ideally this belongs in Iceberg >>> (and if Iceberg and Delta Lake can agree on how to do it that would be >

Re: [Early Feedback] Variant and Subcolumnarization Support

it may be >>> better for the more broad community if it was a sub project. As a >>> sub-project I would still prefer it being an Iceberg Subproject since we >>> are engine/file-format agnostic. >>> >>> 3. I support adding just Variant. >>> >>

Re: [Early Feedback] Variant and Subcolumnarization Support

:32:22 Aihua Xu wrote: > Thanks for the discussion. > > I will move forward to work on spec PR. > > Regarding the implementation, we will have module for Variant support in > Iceberg so we will not have to bring in Spark libraries. > > I'm reposting the meeting inv

Re: [Early Feedback] Variant and Subcolumnarization Support

think about a Java implementation in Iceberg? > >>>>>>>> > >>>>>>>> Ryan > >>>>>>>> > >>>>>>>> On Fri, Jul 12, 2024 at 12:28 PM Ryan Blue > >>>>>>>> wrote: > &

Re: [Early Feedback] Variant and Subcolumnarization Support

rewrite variant > data. (Also, note that I work at Databricks and have an interest in > increasing format compatibility.) > > Ryan > > On Thu, Jul 11, 2024 at 11:21 AM Aihua Xu > wrote: > > > [Discuss] Consensus for Variant Encoding > > > > It’s great to

Re: [Early Feedback] Variant and Subcolumnarization Support

632691790 On Tue, May 28, 2024 at 9:21 PM Aihua Xu wrote: > Hello, > > We have drafted the proposal > <https://docs.google.com/document/d/1QjhpG_SVNPZh3anFcpicMQx90ebwjL7rmzFYfUP89Iw/edit> > for Variant data type. Please help review and comment. > > Thanks, > Aihua &g

Re: [Early Feedback] Variant and Subcolumnarization Support

Hello, We have drafted the proposal for Variant data type. Please help review and comment. Thanks, Aihua On Thu, May 16, 2024 at 12:45 PM Jack Ye wrote: > +1 for a JSON/BSON type. We also had the same di