[DISCUSS] Start iceberg-rust 0.3.0 release process

Hello, everyone I'm starting this thread to discuss initiating the release process for iceberg-rust 0.3.0. We had planned to start this release months ago but were blocked by the Avro Rust release. Now that this has been resolved, and most issues in our 0.3.0 milestone have been addressed, I b

Re: Welcome Péter, Amogh and Eduard to the Apache Iceberg PMC

Congrats! Thanks for your contribution. On Wed, Aug 14, 2024, at 11:32, Renjie Liu wrote: > Congratulations, everyone! > > On Wed, Aug 14, 2024 at 11:14 AM roryqi wrote: >> Congrats! >> >> Steven Wu 于2024年8月14日周三 11:12写道： >>> Congratulations! >>> >>> On Tue, Aug 13, 2024 at 7:18 PM Kevin Liu

Re: Welcome Péter, Amogh and Eduard to the Apache Iceberg PMC

Congratulations, everyone! On Wed, Aug 14, 2024 at 11:14 AM roryqi wrote: > Congrats! > > Steven Wu 于2024年8月14日周三 11:12写道： > >> Congratulations! >> >> On Tue, Aug 13, 2024 at 7:18 PM Kevin Liu wrote: >> >>> Congratulations all! >>> >>> On Wed, Aug 14, 2024 at 9:38 AM Jun H. wrote: >>> Co

Re: Welcome Péter, Amogh and Eduard to the Apache Iceberg PMC

Congrats! Steven Wu 于2024年8月14日周三 11:12写道： > Congratulations! > > On Tue, Aug 13, 2024 at 7:18 PM Kevin Liu wrote: > >> Congratulations all! >> >> On Wed, Aug 14, 2024 at 9:38 AM Jun H. wrote: >> >>> Congratulations! >>> >>> >>> On Tue, Aug 13, 2024 at 4:28 PM Rodrigo Meneses >>> wrote: >>> >

Re: Welcome Péter, Amogh and Eduard to the Apache Iceberg PMC

Congratulations! On Tue, Aug 13, 2024 at 7:18 PM Kevin Liu wrote: > Congratulations all! > > On Wed, Aug 14, 2024 at 9:38 AM Jun H. wrote: > >> Congratulations! >> >> >> On Tue, Aug 13, 2024 at 4:28 PM Rodrigo Meneses >> wrote: >> >>> This is amazing! Congratulations!!! >>> >>> On Tue, Aug 13,

Re: [VOTE] Merge REST spec clarification on how servers should handle unknown updates/requirements

+1On Aug 14, 2024, at 2:24 AM, Ryan Blue wrote:+1On Tue, Aug 13, 2024 at 8:59 AM Yufei Gu wrote:+1YufeiOn Tue, Aug 13, 2024 at 8:57 AM Eduard Tudenhöfner wrote:+1On Tue, Aug 13, 2024 at 5:09 PM Amogh Jahagirdar <2am...@gmail.com> wrote:I've opened

[VOTE] Release Apache PyIceberg 0.7.1rc2

Hi Everyone, I propose that we release the following RC as the official PyIceberg 0.7.1 release. A summary of the high level features: * Fix `delete` to trace existing manifests when a data file is partially rewritten * Fix 'to_arrow_batch_rea

Re: Welcome Péter, Amogh and Eduard to the Apache Iceberg PMC

This is amazing! Congratulations!!! On Tue, Aug 13, 2024 at 3:07 PM karuppayya wrote: > Congratulations, everyone! > > On Tue, Aug 13, 2024 at 2:54 PM Sung Yun wrote: > >> Congratulations Péter, Amogh and Eduard! >> >> On 2024/08/13 20:25:17 Russell Spitzer wrote: >> > Hi Y'all, >> > >> > It is

Re: Welcome Péter, Amogh and Eduard to the Apache Iceberg PMC

Congratulations, everyone! On Tue, Aug 13, 2024 at 2:54 PM Sung Yun wrote: > Congratulations Péter, Amogh and Eduard! > > On 2024/08/13 20:25:17 Russell Spitzer wrote: > > Hi Y'all, > > > > It is my pleasure to let everyone know that the Iceberg PMC has voted to > > have several talented individ

Re: Welcome Péter, Amogh and Eduard to the Apache Iceberg PMC

Congratulations Péter, Amogh and Eduard! On 2024/08/13 20:25:17 Russell Spitzer wrote: > Hi Y'all, > > It is my pleasure to let everyone know that the Iceberg PMC has voted to > have several talented individuals join us. > > So without further ado, please welcome Péter Váry, Amogh Jahagirdar an

Re: [DISCUSS] Changing namespace separator in REST spec

It seems like our specification made a poor choice in terms of separator character, do we have any disagreement on this point? Not from me! But I think the next question is whether there is a better one for the general case so that we can just choose a new one and be done. I can’t think of a bette

Re: Welcome Péter, Amogh and Eduard to the Apache Iceberg PMC

Congrats, everyone :) On Tue, Aug 13, 2024 at 4:25 PM Russell Spitzer wrote: > Hi Y'all, > > It is my pleasure to let everyone know that the Iceberg PMC has voted to > have several talented individuals join us. > > So without further ado, please welcome Péter Váry, Amogh Jahagirdar and > Eduard

Re: Welcome Péter, Amogh and Eduard to the Apache Iceberg PMC

Congrats folks! Cheers, Matyas On Tue, Aug 13, 2024 at 1:25 PM Russell Spitzer wrote: > Hi Y'all, > > It is my pleasure to let everyone know that the Iceberg PMC has voted to > have several talented individuals join us. > > So without further ado, please welcome Péter Váry, Amogh Jahagirdar and

Re: Welcome Péter, Amogh and Eduard to the Apache Iceberg PMC

Congrats 👏 On Wed, 14 Aug 2024 at 07:15, Yufei Gu wrote: > Congratulations, Peter, Amogh and Eduard! Well deserved! > > Yufei > > > On Tue, Aug 13, 2024 at 2:03 PM John Zhuge wrote: > >> Congratulations, everyone! >> >> On Tue, Aug 13, 2024 at 1:59 PM huaxin gao >> wrote: >> >>> Congratulatio

Re: Welcome Péter, Amogh and Eduard to the Apache Iceberg PMC

Congratulations, Peter, Amogh and Eduard! Well deserved! Yufei On Tue, Aug 13, 2024 at 2:03 PM John Zhuge wrote: > Congratulations, everyone! > > On Tue, Aug 13, 2024 at 1:59 PM huaxin gao wrote: > >> Congratulations, everyone! >> >> On Tue, Aug 13, 2024 at 1:53 PM Ryan Blue >> wrote: >> >>>

Re: Welcome Péter, Amogh and Eduard to the Apache Iceberg PMC

Congratulations, everyone! On Tue, Aug 13, 2024 at 2:03 PM John Zhuge wrote: > Congratulations, everyone! > > On Tue, Aug 13, 2024 at 1:59 PM huaxin gao wrote: > >> Congratulations, everyone! >> >> On Tue, Aug 13, 2024 at 1:53 PM Ryan Blue >> wrote: >> >>> Congratulations! Thanks for all your

Re: Welcome Péter, Amogh and Eduard to the Apache Iceberg PMC

Congratulations, everyone! On Tue, Aug 13, 2024 at 1:59 PM huaxin gao wrote: > Congratulations, everyone! > > On Tue, Aug 13, 2024 at 1:53 PM Ryan Blue > wrote: > >> Congratulations! Thanks for all your contributions! >> >> On Tue, Aug 13, 2024 at 1:48 PM Steve Zhang >> wrote: >> >>> Congrats

Re: Welcome Péter, Amogh and Eduard to the Apache Iceberg PMC

Congratulations, everyone! On Tue, Aug 13, 2024 at 1:53 PM Ryan Blue wrote: > Congratulations! Thanks for all your contributions! > > On Tue, Aug 13, 2024 at 1:48 PM Steve Zhang > wrote: > >> Congrats everyone, well deserved! >> >> Thanks, >> Steve Zhang >> >> >> >> On Aug 13, 2024, at 1:31 PM,

Re: Welcome Péter, Amogh and Eduard to the Apache Iceberg PMC

Congratulations! Thanks for all your contributions! On Tue, Aug 13, 2024 at 1:48 PM Steve Zhang wrote: > Congrats everyone, well deserved! > > Thanks, > Steve Zhang > > > > On Aug 13, 2024, at 1:31 PM, Bill Zhang > wrote: > > Congratulations everyone. > > On Tue, Aug 13, 2024 at 1:28 PM Szehon

Re: Welcome Péter, Amogh and Eduard to the Apache Iceberg PMC

Congrats everyone, well deserved! Thanks, Steve Zhang > On Aug 13, 2024, at 1:31 PM, Bill Zhang wrote: > > Congratulations everyone. > > On Tue, Aug 13, 2024 at 1:28 PM Szehon Ho > wrote: >> Congratulations all, very well deserved! >> >> Thanks >> Szehon >>

Re: Welcome Péter, Amogh and Eduard to the Apache Iceberg PMC

Congratulations everyone. On Tue, Aug 13, 2024 at 1:28 PM Szehon Ho wrote: > Congratulations all, very well deserved! > > Thanks > Szehon > > On Tue, Aug 13, 2024 at 10:25 PM Russell Spitzer < > russell.spit...@gmail.com> wrote: > >> Hi Y'all, >> >> It is my pleasure to let everyone know that th

Re: Welcome Péter, Amogh and Eduard to the Apache Iceberg PMC

Congratulations all, very well deserved! Thanks Szehon On Tue, Aug 13, 2024 at 10:25 PM Russell Spitzer wrote: > Hi Y'all, > > It is my pleasure to let everyone know that the Iceberg PMC has voted to > have several talented individuals join us. > > So without further ado, please welcome Péter V

Welcome Péter, Amogh and Eduard to the Apache Iceberg PMC

Hi Y'all, It is my pleasure to let everyone know that the Iceberg PMC has voted to have several talented individuals join us. So without further ado, please welcome Péter Váry, Amogh Jahagirdar and Eduard Tudenhoefner to the Apache Iceberg PMC. As usual I am excited about the future of this comm

Re: [DISCUSS] Materialized Views: Lineage and State information

Hi Everyone, Just a follow up on this thread. Thanks Benny and Micah for the discussion on the doc [1]. We have been converging more on using UUIDs from the discussion. The only open question was related to UUIDs (of underlying views/tables) being stale upon a REPLACE (or DROP and CREATE) operatio

Re: [VOTE] Merge REST spec clarification on how servers should handle unknown updates/requirements

+1 On Tue, Aug 13, 2024 at 8:59 AM Yufei Gu wrote: > +1 > Yufei > > > On Tue, Aug 13, 2024 at 8:57 AM Eduard Tudenhöfner < > etudenhoef...@apache.org> wrote: > >> +1 >> >> On Tue, Aug 13, 2024 at 5:09 PM Amogh Jahagirdar <2am...@gmail.com> >> wrote: >> >>> I've opened a PR [1] to clarify in the

Re: [DISCUSS] Variant Spec Location

Thanks Russell for bringing this up. This is the main blocker to move forward with the Variant support in Iceberg and hopefully we can have a consensus. To me, I also feel it makes more sense to move the spec into Iceberg rather than Spark engine owns it and we try to keep it compatible with Spark

Re: [VOTE] Merge REST spec clarification on how servers should handle unknown updates/requirements

+1 Yufei On Tue, Aug 13, 2024 at 8:57 AM Eduard Tudenhöfner wrote: > +1 > > On Tue, Aug 13, 2024 at 5:09 PM Amogh Jahagirdar <2am...@gmail.com> wrote: > >> I've opened a PR [1] to clarify in the REST spec that if a server >> receives an unknown update or requirement as part of any the commit >>

Re: [VOTE] Merge REST spec clarification on how servers should handle unknown updates/requirements

+1 On Tue, Aug 13, 2024 at 5:09 PM Amogh Jahagirdar <2am...@gmail.com> wrote: > I've opened a PR [1] to clarify in the REST spec that if a server > receives an unknown update or requirement as part of any the commit > endpoints, that the server must fail with a 400 error response. > > Please vote

Re: [DISCUSS] Changing namespace separator in REST spec

I feel like we are over complicating this situation. It seems like our specification made a poor choice in terms of separator character, do we have any disagreement on this point? It looks like by choosing a control character, we ended up generating requests which modern server systems define as p

[VOTE] Merge REST spec clarification on how servers should handle unknown updates/requirements

I've opened a PR [1] to clarify in the REST spec that if a server receives an unknown update or requirement as part of any the commit endpoints, that the server must fail with a 400 error response. Please vote on merging this change. The vote will remain open for at least 72 hours. [] +1 [] +0 [

Re: [DISCUSS] Clarify in REST spec expected implementation behavior for unknown updates or requirements

Thanks all for your feedback! Since it seems like there's general consensus and this discussion thread has been open for quite some time at this point, I will start a vote thread on this. Thank you, Amogh Jahagirdar On Wed, Aug 7, 2024 at 7:17 PM Yufei Gu wrote: > Thanks Amogh for starting thi

Re: [DISCUSS] Filesystem in PyIceberg

I believe that now I understand how to leverage the metadata tables to deal with removing orphan files. I didn't know that the DELETE_FILES metadata table existed, so I believe this is what Fokko meant. Fokko, was your idea to use the DELETE_FILES and ALL_FILES metadata tables? Do you know why

Re: [DISCUSS] Filesystem in PyIceberg

On Tue, 13 Aug 2024 at 03:50, Xuanwo wrote: > Hi, André > > Thanks a lot for starting this thread. > > List operations on storage services are expensive and slow. That's why > Iceberg is designed to store metadata in files and avoid using list > operations in FileIO. However, `orphan file removal

Re: [DISCUSS] Changing namespace separator in REST spec

I do not see why "old clients are already broken" - they behave according to the REST spec. The current REST spec states "If parent is a multipart namespace, the parts must be separated by the unit separator (`0x1F`) byte" [1]. Using a different separator, even if it is "backwards compatible",