Re: [VOTE] Release Apache Iceberg 1.9.1 RC1

2025-05-27 Thread Aihua Xu
+1 (non-binding). Verified against Snowflake engine. On Tue, May 27, 2025 at 9:28 AM huaxin gao wrote: > +1 (non-binding) > Verified signature, checksum, license and ran some tests. > > On Tue, May 27, 2025 at 9:06 AM Russell Spitzer > wrote: > >> For all those who haven't seen this before, GP

Re: [VOTE] Release Apache Iceberg 1.9.1 RC0

2025-05-21 Thread Aihua Xu
from newer clients? >> >>> >> >>> So in this case, newer clients are trying to express deletions with >> larger sizes and the server is unable to handle it due to the assertion in >> the older implementation, not because the protocol changed. Though I c

Re: [VOTE] Adopt the v3 spec changes

2025-05-19 Thread Aihua Xu
+1 (non-binding). Great to see we are moving forward. On Mon, May 19, 2025 at 9:06 PM Denny Lee wrote: > +1 (non-binding) > > On Mon, May 19, 2025 at 9:04 PM Yufei Gu wrote: > >> +1 (binding) >> >> Yufei >> >> >> On Mon, May 19, 2025 at 8:59 PM Amogh Jahagirdar <2am...@gmail.com> >> wrote: >>

Re: [VOTE] Release Apache Iceberg 1.9.1 RC0

2025-05-19 Thread Aihua Xu
d and can get the benefits of the > change in the 1.9 release. > > If we think it's simpler to just revert for 1.9 and cycle these proposed > server changes for 1.10 and then the client changes for the release after > 1.10, I think I'm OK (+0) with that as well. > > Than

Re: Spark 4.0/Iceberg Integration Merged – Spark 3.5 Merges Can Resume

2025-05-19 Thread Aihua Xu
Thanks a lot to get that merged! Huaxin. On Fri, May 16, 2025 at 12:05 PM Amogh Jahagirdar <2am...@gmail.com> wrote: > Thanks Huaxin for your patience while getting this through! > > On Fri, May 16, 2025 at 12:20 PM Steve wrote: > >> Thanks Huaxin! Great work and looking forward to the spark 4.0

Re: [VOTE] Release Apache Iceberg 1.9.1 RC0

2025-05-19 Thread Aihua Xu
assertion in >>> the older implementation, not because the protocol changed. Though I can >>> see the grey area in that it either forces clients to not upgrade for Java >>> server implementations which haven't upgraded OR it server implementations >>> end up

Re: [VOTE] Release Apache Iceberg 1.9.1 RC0

2025-05-19 Thread Aihua Xu
I have verified RC against Snowflake build. Everything works except one issue introduced by https://github.com/apache/iceberg/pull/12670/ : the client with 1.9.x can't work with the catalog server with old library to remove the snapshots since the the client now will remove the snapshots in bulk w

Re: [Discuss] Iceberg 1.9.1 Release

2025-05-17 Thread Aihua Xu
Is https://github.com/apache/iceberg/pull/12670/ considered introducing a broken change for catalog in 1.9.0? Now the client can send a list of snapshots to the server to remove in TableMetadata class while the server with the old Iceberg library still checks to make sure only a single snapshotId i

Re: [Discuss] spec: Add missing 'spec_id' for data_file

2025-05-05 Thread Aihua Xu
Szehon pointed out that it's not serialized to the manifest file. So it's used internally but it doesn't need to be in the spec. Thanks, Aihua On Mon, May 5, 2025 at 10:34 AM Aihua Xu wrote: > Hi all, > > I notice that we are missing spec_id for data_file in the sp

[Discuss] spec: Add missing 'spec_id' for data_file

2025-05-05 Thread Aihua Xu
Hi all, I notice that we are missing spec_id for data_file in the spec while working on V3 features (https://github.com/apache/iceberg/pull/12970) and it seems obvious. But let me know if a vote is needed for such change. Thanks, Aihua

Re: [VOTE] Add encryption keys to table metadata

2025-04-30 Thread Aihua Xu
+1 (non-binding). On Wed, Apr 30, 2025 at 3:49 PM Daniel Weeks wrote: > +1 (binding) > > On Wed, Apr 30, 2025 at 1:38 PM Amogh Jahagirdar <2am...@gmail.com> wrote: > >> +1 (binding) >> >> On Wed, Apr 30, 2025 at 1:29 PM Anurag Mantripragada >> wrote: >> >>> +1 (non-binding) >>> >>> ~ Anurag Man

Re: [VOTE] Release Apache Iceberg 1.9.0 RC2

2025-04-26 Thread Aihua Xu
+1 (non-binding) 1. Run the tests 2. Test against Snowflake build Thanks Aihua On Sat, Apr 26, 2025 at 7:02 AM Amogh Jahagirdar <2am...@gmail.com> wrote: > +1 (binding) > > 1. Verified sigs/checksum/license > 2. Ran build/test with JDK 17 > > Thanks Ajantha for running the release! > > On Fri,

Re: [VOTE] Spec Update: Variant Field Lower/Upper Bounds

2025-04-21 Thread Aihua Xu
This passes with 13 votes (+6 bindings and +7 non-bindings) and no -1 or +0 votes. Thanks everyone! On Mon, Apr 21, 2025 at 1:26 PM Aihua Xu wrote: > +1 (non-binding) from me. > > On Sun, Apr 20, 2025 at 12:19 PM Denny Lee wrote: > >> +1 (non-binding) >> >> On Su

Re: [VOTE] Spec Update: Variant Field Lower/Upper Bounds

2025-04-21 Thread Aihua Xu
+1 (non-binding) from me. On Sun, Apr 20, 2025 at 12:19 PM Denny Lee wrote: > +1 (non-binding) > > On Sun, Apr 20, 2025 at 11:22 Fokko Driesprong wrote: > >> +1 (binding) >> >> Op zo 20 apr 2025 om 05:59 schreef Daniel Weeks : >> >>> +1 (binding) >>> >>> On Sat, Apr 19, 2025, 12:07 AM Gang Wu

[VOTE] Spec Update: Variant Field Lower/Upper Bounds

2025-04-18 Thread Aihua Xu
Hi all, I'd like to initiate a vote to include a spec update for supporting lower and upper bounds on Variant fields. Summary of the change: The writer determines which fields to collect bounds for a Variant column. Field bounds are stored as serialized Variant objects, where each key is a normal

Re: [VOTE] Update row lineage spec ID assignment

2025-04-17 Thread Aihua Xu
+ (non-binding). On Thu, Apr 17, 2025 at 11:22 AM Steven Wu wrote: > +1 (binding) > > On Thu, Apr 17, 2025 at 11:09 AM Amogh Jahagirdar <2am...@gmail.com> > wrote: > >> +1 (binding) >> >> On Thu, Apr 17, 2025 at 11:54 AM Szehon Ho >> wrote: >> >>> +1 (binding) Seems cleaner to me. >>> >>> Than

Re: [VOTE] Release Apache Iceberg 1.9.0 RC0

2025-04-13 Thread Aihua Xu
+1 (non-binding). Verified against Snowflake build. On Sun, Apr 13, 2025 at 12:20 AM Yuya Ebihara < yuya.ebih...@starburstdata.com> wrote: > +1 (non-binding) > > Trino Iceberg tests with 1.9.0 RC0 all passed except for the V3 write > test, which failed as expected since row lineage is now requir

Variant bound spec

2025-03-26 Thread Aihua Xu
Hi everyone, I'm updating the spec to define the lower/upper bounds for shredded Variant PR#12658 . Basically we are using a Variant object to hold the bound value for each shredded subcolumn as discussed in the community sync. Let me know if you have

Re: [VOTE] Release Apache Iceberg 1.8.1 RC1

2025-02-27 Thread Aihua Xu
ding) from me as well. > > Thanks, > > Alex > > On Thu, Feb 27, 2025 at 6:45 PM Aihua Xu wrote: > >> Hi Alex, >> >> I checked our code further. We have an internal implementation >> for RESTClient which needs an update after your change to place the token

Re: [VOTE] Release Apache Iceberg 1.8.1 RC1

2025-02-27 Thread Aihua Xu
change to get the token from the headers of DefaultAuthSession. With that, +1 (non-binding) Thanks, Aihua On Thu, Feb 27, 2025 at 9:26 AM Alex Dutra wrote: > Hi Aihua Xu, > > I reviewed your PR but without further details I do not agree with your > change, and I am unable to reprodu

Re: [VOTE] Release Apache Iceberg 1.8.1 RC1

2025-02-27 Thread Aihua Xu
gt; the OAuth2 server URI configuration and defaults to > http://127.0.0.1:8181/api/catalog/v1/oauth/tokens. This automatic > fallback will be removed in a future Iceberg release.It is recommended to > configure the OAuth2 endpoint using the 'oauth2-server-uri' property to be > prepa

Re: [VOTE] Release Apache Iceberg 1.8.1 RC1

2025-02-26 Thread Aihua Xu
I tested 1.8.1 RC with Snowflake build. I'm seeing the following (I'm not seeing that in 1.7.x). "exception": "java.io.IOException: *Authorization header is missing*\n\tat org.apache.polaris.service.dropwizard.auth.PolarisPrincipalAuthenticator.filter(PolarisPrincipalAuthenticator.java:43)\n\tat o

Re: [VOTE] Allow Row-Lineage with Equality Deletes

2025-02-20 Thread Aihua Xu
+1 (non-binding). On Thu, Feb 20, 2025 at 9:41 AM Huang-Hsiang Cheng wrote: > +1 (non-binding) > > Thanks, > Huang-Hsiang > > On Feb 20, 2025, at 9:37 AM, huaxin gao wrote: > > +1 (non-binding) > > Thanks Russell! > > On Thu, Feb 20, 2025 at 1:57 AM Fokko Driesprong wrote: > >> +1 >> >> Thanks

Re: [VOTE] Add Geometry and Geography types for V3

2025-02-07 Thread Aihua Xu
+1 (non-binding). On Fri, Feb 7, 2025 at 8:12 AM Jean-Baptiste Onofré wrote: > +1 > > That's a great progress ! Thanks ! > > Regards > JB > > On Thu, Feb 6, 2025 at 9:01 PM Szehon Ho wrote: > > > > Hi everyone > > > > We would like to add Geometry and Geography types to the Iceberg V3 spec: > >

Re: Welcome Huaxin Gao as a committer!

2025-02-07 Thread Aihua Xu
Congrats Huaxin!. On Fri, Feb 7, 2025 at 1:35 AM Manish Malhotra < manish.malhotra.w...@gmail.com> wrote: > Congratulations Huaxin! > > On Thu, Feb 6, 2025 at 6:34 PM Ron Liu wrote: > >> Congrats >> >> Best, >> Ron >> >> roryqi 于2025年2月7日周五 10:21写道: >> >>> Congrats! >>> >>> Anton Okolnychyi 于2

Re: guideline for interface change

2025-02-03 Thread Aihua Xu
t; Fokko > > Op zo 2 feb 2025 om 21:12 schreef Aihua Xu : > >> Thanks folks for suggestions. I will keep the existing one and mark it as >> deprecated just in case it's getting used elsewhere. I will add a new one. >> >> On Sat, Feb 1, 2025 at 2:40 AM Russell

Re: guideline for interface change

2025-02-02 Thread Aihua Xu
>> the new method using the old one? >> >> This would keep the old functionality until the deprecated method is >> removed. >> >> On Sat, Feb 1, 2025, 02:01 Aihua Xu wrote: >> >>> Hi folks, >>> >>> What is the general guideline

guideline for interface change

2025-01-31 Thread Aihua Xu
Hi folks, What is the general guideline for interface change? I'm trying to change PrimitiveType Types::fromPrimitiveString() => Type Types::fromTypeString() in https://github.com/apache/iceberg/pull/11831/files#diff-736caed551a388d34b08f223954ae7ecb2fdac9d90a4098ceedd95207d7efd4dR1149-R1152 to

Re: [VOTE] Add Variant type to Iceberg Spec

2025-01-30 Thread Aihua Xu
Thanks Russell for following up on this. On Wed, Jan 29, 2025 at 2:50 PM Russell Spitzer wrote: > Given that I think we are all in agreement, and Micah has signed off on > the PR I think we can go ahead and merge this one. > > On Tue, Dec 10, 2024 at 12:16 AM Aihua Xu wrote: > &

Re: [DISCUSS/VOTE] Add in ChangeLog Reserved Field IDs to Spec and Decrement Row Lineage Reserved IDs

2025-01-26 Thread Aihua Xu
+1 (non-binding). Thanks for fixing it. On Sun, Jan 26, 2025 at 11:30 AM Anton Okolnychyi wrote: > +1 good catch > > нд, 26 січ. 2025 р. о 08:33 Steve Zhang > пише: > >> +1 (non-binding) >> >> Thanks, >> Steve Zhang >> >> >> >> On Jan 25, 2025, at 10:48 AM, huaxin gao wrote: >> >> +1 (non-bin

Re: [DISCUSS, VOTE] OpenAPI Metadata Update for EnableRowLineage

2025-01-23 Thread Aihua Xu
+ (non binding). Thanks Russell. On Thu, Jan 23, 2025 at 2:05 AM Jean-Baptiste Onofré wrote: > +1 (non binding) > > Regards > JB > > On Wed, Jan 22, 2025 at 11:51 PM Russell Spitzer > wrote: > > > > Hey Y'all > > > > Yet another Row Lineage Spec update. This adds a MetadataUpdate > EnableRowLi

Re: [VOTE] Document Snapshot Summary Optional Fields as Subsection of Appendix F in Spec

2025-01-22 Thread Aihua Xu
+1 non-binding. On Wed, Jan 22, 2025 at 7:57 AM Kevin Liu wrote: > +1 non-binding > > Third time's the charm! :) > > On Wed, Jan 22, 2025 at 4:44 AM Jean-Baptiste Onofré > wrote: > >> +1 (non binding) >> >> Regards >> JB >> >> On Tue, Jan 21, 2025 at 9:19 PM Honah J. wrote: >> > >> > Hi everyo

[Proposal] Iceberg Metadata for Variant Shredding

2025-01-13 Thread Aihua Xu
Hi all, I'm trying to propose the metadata change for variant shredding in ( https://docs.google.com/document/d/1gAvt0x_ez89O8y-YqkCdMnTEykb-583YslYOgzf5sPg/edit?tab=t.0) and https://github.com/apache/iceberg/issues/11958. Basically, in Iceberg format V3, we collect subcolumn lower/upper bounds i

Re: [VOTE] Add Variant type to Iceberg Spec

2024-12-09 Thread Aihua Xu
that scenario >>> is unlikely) >>> 3. There is very little in our change set here that specifically >>> references the Parquet spec except for our reference link to it. >>> >>> I don't think there is anything that will happen in the spec that

Re: [DISCUSS] Apache Iceberg Summit 2025 - Selection Committee

2024-12-09 Thread Aihua Xu
Hi JB, I would love to help. Thanks, Aihua On Tue, Dec 3, 2024 at 4:37 AM Anurag Mantripragada wrote: > Thanks for the email. I’m happy to help as well. > > > Anurag Mantripragada > > On Dec 1, 2024, at 7:27 PM, Nick Riasanovsky wrote: > > Happy to volunteer as well. > > - Nick Riasanovsky >

[VOTE] Add Variant type to Iceberg Spec

2024-11-22 Thread Aihua Xu
Hi everyone, I've updated the Iceberg spec to include the new Variant type as part of #10831 . The changes are basically complete. This is a heads-up about the upcoming change. Please review and +1 to acknowledge, so we will merge. Thanks, Aihua

Re: [VOTE] Deprecate and remove last-column-id

2024-11-20 Thread Aihua Xu
+1 non-binding. Thanks for driving this. On Tue, Nov 19, 2024 at 5:50 PM Renjie Liu wrote: > +1, thanks Fokko! > > On Wed, Nov 20, 2024 at 8:45 AM Steve Zhang > wrote: > >> +1 nb >> >> Thanks, >> Steve Zhang >> >> >> >> On Nov 19, 2024, at 12:18 AM, Fokko Driesprong wrote: >> >> Hi everyone,

Re: [DISCUSS] Iceberg 1.7.1 release

2024-11-15 Thread Aihua Xu
That makes sense. Originally I thought wasb scheme change was a feature change. I will wait for 1.8.0. then. On Fri, Nov 15, 2024 at 7:18 AM Bryan Keller wrote: > I agree, we should leave this one for 1.8. > -Bryan > > On Nov 14, 2024, at 9:22 PM, Aihua Xu > wrote: > >

Re: [DISCUSS] Iceberg 1.7.1 release

2024-11-14 Thread Aihua Xu
Hi Bryan, I would like to include the following in 1.7.1 if possible. I keep the change minimum and I will try to get this merged soon. Add Variant data type to API: https://github.com/apache/iceberg/pull/11324 Thanks, Aihua On Thu, Nov 14, 2024 at 11:21 AM Bryan Keller wrote: > Hi all, > > A

Re: [VOTE] Release Apache Iceberg 1.7.0 RC1

2024-11-08 Thread Aihua Xu
+1 (non-binding) Verified signature, checksums, license Check on Spark with some manual testing Run some unit tests and integration tests on Snowflake Thanks Russell to drive this. Aihua

Bayarea Iceberg meetup in November

2024-10-03 Thread Aihua Xu
Hi community! The Apache Iceberg community is gathering in San Francisco on November 4th. Whether you’re interested in presenting or just want to join us for some networking, you can find the event details and RSVP here: https://lu.ma/fholq6oz . Hope to see you there! Thanks, Aihua

Re: [DISCUSS] Variant Spec Location

2024-08-28 Thread Aihua Xu
Le Dem wrote: > Thank you Gang, that's sounds like a good idea to me as well > > On Fri, Aug 23, 2024 at 8:47 AM Aihua Xu > wrote: > >> Thanks Gang for initiating the discussion. >> >> On Fri, Aug 23, 2024 at 2:22 AM Gang Wu wrote: >> >>

Re: [DISCUSS] Variant Spec Location

2024-08-23 Thread Aihua Xu
Thanks Gang for initiating the discussion. On Fri, Aug 23, 2024 at 2:22 AM Gang Wu wrote: > Thanks Aihua! > > I've started the discussion in dev@parquet: > https://lists.apache.org/thread/6h58hj39lhqtcyd2hlsyvqm4lzdh4b9z > > Best, > Gang > > On Fri, Aug 23,

Re: [DISCUSS] Variant Spec Location

2024-08-22 Thread Aihua Xu
gt; collaborate >> >>>> in >> >>>>>>> doing that. One thing that I don't want to do is have the Iceberg >> >>>>> project >> >>>>>>> stay in a holding pattern without any clear road

Re: [DISCUSS] Variant Spec Location

2024-08-13 Thread Aihua Xu
Thanks Russell for bringing this up. This is the main blocker to move forward with the Variant support in Iceberg and hopefully we can have a consensus. To me, I also feel it makes more sense to move the spec into Iceberg rather than Spark engine owns it and we try to keep it compatible with Spark

Bayarea Apache Iceberg meeting - Sept. 2024

2024-07-31 Thread Aihua Xu
Hi community, We're thrilled to announce an upcoming Apache Iceberg Community Meetup in the Bay Area! This is a fantastic opportunity to connect with fellow enthusiasts, share insights, and dive into the latest developments in the Apache Iceberg ecosystem. 📅 Date: September 5th, 2024 ⏰ Time: 5:0

Re: [Early Feedback] Variant and Subcolumnarization Support

2024-07-29 Thread Aihua Xu
general, and I agree we should avoid >> significantly complicating the design for an uncommon use case. I would >> like to understand the implication for operations like file compaction. Is >> it now up to this engine's discretion for how to combine files and if >> deci

Re: [Early Feedback] Variant and Subcolumnarization Support

2024-07-26 Thread Aihua Xu
f not modifying >> decimal scales, but I'm curious if this distinction satisfies everyone. If >> we remove the wording from the proposal that recommends modifying decimals >> and leave this to the engine, do we have agreement? >> >> On Thu, Jul 25, 2024 at 6:46 PM Aihua X

Re: [Early Feedback] Variant and Subcolumnarization Support

2024-07-25 Thread Aihua Xu
at 10:14 AM Aihua Xu wrote: > Yes. This time I was able to record it and I will share it when it’s > processed. > > > On Jul 25, 2024, at 10:01 AM, Amogh Jahagirdar <2am...@gmail.com> wrote: > >  > Any chance this meeting was recorded? I couldn't make it

Re: [Early Feedback] Variant and Subcolumnarization Support

2024-07-25 Thread Aihua Xu
irdarOn Tue, Jul 23, 2024 at 11:30 AM Aihua Xu <aihu...@gmail.com> wrote:Thanks folks for additional discussion.There are some questions related to subcolumniziation (spark shredding - see the discussion) and we would like to host another meeting to mainly discuss that since we plan to adop

Re: [Early Feedback] Variant and Subcolumnarization Support

2024-07-23 Thread Aihua Xu
>>> a superset of the Iceberg type system (UUID is supported as a canonical >>> extension type [2]). >>> >>> For point 4 subcolumnarization, I think ideally this belongs in Iceberg >>> (and if Iceberg and Delta Lake can agree on how to do it that would be >

Re: [Early Feedback] Variant and Subcolumnarization Support

2024-07-20 Thread Aihua Xu
it may be >>> better for the more broad community if it was a sub project. As a >>> sub-project I would still prefer it being an Iceberg Subproject since we >>> are engine/file-format agnostic. >>> >>> 3. I support adding just Variant. >>> >>

Re: [Early Feedback] Variant and Subcolumnarization Support

2024-07-17 Thread Aihua Xu
:32:22 Aihua Xu wrote: > Thanks for the discussion. > > I will move forward to work on spec PR. > > Regarding the implementation, we will have module for Variant support in > Iceberg so we will not have to bring in Spark libraries. > > I'm reposting the meeting inv

Re: [Early Feedback] Variant and Subcolumnarization Support

2024-07-15 Thread Aihua Xu
think about a Java implementation in Iceberg? > >>>>>>>> > >>>>>>>> Ryan > >>>>>>>> > >>>>>>>> On Fri, Jul 12, 2024 at 12:28 PM Ryan Blue > >>>>>>>> wrote: > &

Re: [Early Feedback] Variant and Subcolumnarization Support

2024-07-12 Thread Aihua Xu
rewrite variant > data. (Also, note that I work at Databricks and have an interest in > increasing format compatibility.) > > Ryan > > On Thu, Jul 11, 2024 at 11:21 AM Aihua Xu > wrote: > > > [Discuss] Consensus for Variant Encoding > > > > It’s great to

Re: [Early Feedback] Variant and Subcolumnarization Support

2024-07-11 Thread Aihua Xu
632691790 On Tue, May 28, 2024 at 9:21 PM Aihua Xu wrote: > Hello, > > We have drafted the proposal > <https://docs.google.com/document/d/1QjhpG_SVNPZh3anFcpicMQx90ebwjL7rmzFYfUP89Iw/edit> > for Variant data type. Please help review and comment. > > Thanks, > Aihua &g

Re: [Early Feedback] Variant and Subcolumnarization Support

2024-05-31 Thread Aihua Xu
Hello, We have drafted the proposal for Variant data type. Please help review and comment. Thanks, Aihua On Thu, May 16, 2024 at 12:45 PM Jack Ye wrote: > +1 for a JSON/BSON type. We also had the same di