Re: [DISCUSS] Start iceberg-rust 0.3.0 release process

Thanks Xuanwo for driving this, very excited to see this happening. Let me know if there is anything I can help with! Kind regards, Fokko Op wo 14 aug 2024 om 08:58 schreef Xuanwo : > Hello, everyone > > I'm starting this thread to discuss initiating the release process for > iceberg-rust 0.3.0.

Re: Welcome Péter, Amogh and Eduard to the Apache Iceberg PMC

Congratulations and welcome! Kind regards, Fokko Op wo 14 aug 2024 om 06:23 schreef Xuanwo : > Congrats! Thanks for your contribution. > > On Wed, Aug 14, 2024, at 11:32, Renjie Liu wrote: > > Congratulations, everyone! > > On Wed, Aug 14, 2024 at 11:14 AM roryqi wrote: > > Congrats! > > Steven

Re: Welcome Péter, Amogh and Eduard to the Apache Iceberg PMC

Thanks guys and also congrats to Péter and Amogh! On Wed, Aug 14, 2024 at 9:09 AM Fokko Driesprong wrote: > Congratulations and welcome! > > Kind regards, > Fokko > > Op wo 14 aug 2024 om 06:23 schreef Xuanwo : > >> Congrats! Thanks for your contribution. >> >> On Wed, Aug 14, 2024, at 11:32, Re

[DISCUSS] REST Endpoint discovery

Hey everyone, I'd like to propose a way for REST servers to communicate to clients what endpoints it supports via a new *endpoints* field in the *CatalogConfig* of the *v1/config* endpoint. This enables clients to make better decisions and clearly signal that a particular endpoint isn’t supported

Re: [DISCUSS] Describing REST Server capabilities

@Walaa: Those are all good feedback points that are appreciated. I realized that the capabilities probably add more complexity than necessary. As you mentioned, in the end we really want to know what endpoints a server supports. I've created a new design doc and opened a separate discussion thread

Re: Welcome Péter, Amogh and Eduard to the Apache Iceberg PMC

Thanks everyone, and congratulations to Eduard and Amogh! Eduard Tudenhöfner ezt írta (időpont: 2024. aug. 14., Sze, 9:28): > Thanks guys and also congrats to Péter and Amogh! > > On Wed, Aug 14, 2024 at 9:09 AM Fokko Driesprong wrote: > >> Congratulations and welcome! >> >> Kind regards, >> Fo

Re: [DISCUSS] Changing namespace separator in REST spec

I agree that the simplest approach would be to just pick a new namespace separator while still supporting the old one and be done with it, but this carries the risk that the namespace separator being chosen won't be the right one for everyone. Hence we're making the namespace separator configurabl

Re: [DISCUSS] Variant Spec Location

Thanks Russell and Aihua for pushing Variant support! Given the differences between the supported types and the lack of interest from the other project, I think it is reasonable to duplicate the specification to our repository. I would give very strong emphasis on sticking to the Spark spec as muc

Re: Welcome Péter, Amogh and Eduard to the Apache Iceberg PMC

congratulations all. On Tue, 13 Aug 2024 at 21:25, Russell Spitzer wrote: > Hi Y'all, > > It is my pleasure to let everyone know that the Iceberg PMC has voted to > have several talented individuals join us. > > So without further ado, please welcome Péter Váry, Amogh Jahagirdar and > Eduard Tud

Re: [DISCUSS] Start iceberg-rust 0.3.0 release process

Thanks Xuanwo for driving this! I'm ready to help anytime! On Wed, Aug 14, 2024 at 3:09 PM Fokko Driesprong wrote: > Thanks Xuanwo for driving this, very excited to see this happening. Let me > know if there is anything I can help with! > > Kind regards, > Fokko > > Op wo 14 aug 2024 om 08:58 sc

Re: [DISCUSS] Variant Spec Location

+1 to copy the spec into our repository. I think the best way to keep compatibility is building integration tests. Thanks, Manu On Wed, Aug 14, 2024 at 8:27 PM Péter Váry wrote: > Thanks Russell and Aihua for pushing Variant support! > > Given the differences between the supported types and the

Re: [DISCUSS] Variant Spec Location

+1 to what's already being said here. It is good to copy the spec to Iceberg and add context that's specific to Iceberg, but at the same time, we should maintain compatibility. Kind regards, Fokko Op wo 14 aug 2024 om 15:30 schreef Manu Zhang : > +1 to copy the spec into our repository. I think

Re: Welcome Péter, Amogh and Eduard to the Apache Iceberg PMC

Thanks all, and congrats to Péter and Eduard! On Wed, Aug 14, 2024 at 6:40 AM Steve Loughran wrote: > > congratulations all. > > On Tue, 13 Aug 2024 at 21:25, Russell Spitzer > wrote: > >> Hi Y'all, >> >> It is my pleasure to let everyone know that the Iceberg PMC has voted to >> have several t

Re: [DISCUSS] REST Endpoint discovery

Hi Eduard, How is this proposal related to the Server Capabilities discussion? Thanks, Dmitri. On Wed, Aug 14, 2024 at 5:14 AM Eduard Tudenhöfner wrote: > Hey everyone, > > I'd like to propose a way for REST servers to communicate to clients what > endpoints it supports via a new *endpoints* f

Re: [DISCUSS] REST Endpoint discovery

Would you mind adding a "related proposals" section to the doc to clarify this? Thanks, Dmitri. On Wed, Aug 14, 2024 at 11:15 AM Dmitri Bourlatchkov < dmitri.bourlatch...@dremio.com> wrote: > Hi Eduard, > > How is this proposal related to the Server Capabilities discussion? > > Thanks, > Dmitri.

Re: [Early Feedback] Variant and Subcolumnarization Support

Hello everyone, As it seems the Variant spec decisions are nearly finalized, I would like to ask a clarifying question regarding the difference between SQL Null (missing) and JSON Null. Reading through the Spark specification, source code, and also experimenting with Spark locally, it seems that t

Re: [VOTE] Merge REST spec clarification on how servers should handle unknown updates/requirements

+1 On Tue, Aug 13, 2024 at 7:34 PM xianjin wrote: > +1 > > On Aug 14, 2024, at 2:24 AM, Ryan Blue > wrote: > >  > +1 > > On Tue, Aug 13, 2024 at 8:59 AM Yufei Gu wrote: > >> +1 >> Yufei >> >> >> On Tue, Aug 13, 2024 at 8:57 AM Eduard Tudenhöfner < >> etudenhoef...@apache.org> wrote: >> >>> +1

Re: [VOTE] Merge REST spec clarification on how servers should handle unknown updates/requirements

+1 Thanks for clarifying this Kind regards, Fokko Op wo 14 aug 2024 om 04:34 schreef xianjin : > +1 > > On Aug 14, 2024, at 2:24 AM, Ryan Blue > wrote: > >  > +1 > > On Tue, Aug 13, 2024 at 8:59 AM Yufei Gu wrote: > >> +1 >> Yufei >> >> >> On Tue, Aug 13, 2024 at 8:57 AM Eduard Tudenhöfner <

[DISCUSS] Cleanup svn dev/iceberg

Hi, The dev branch of SVN is used to host artifacts awaiting a vote. It increases the release management burden if this directory has too many unrelated artifacts. For example, there are many old pyiceberg artifacts like: Aiceberg-dev/pyiceberg-0.3.0rc2/pyiceberg-0.3.0.tar.gz Aiceberg-d

Re: [DISCUSS] Cleanup svn dev/iceberg

Hey Xuanwo, Feel free to clean those up as they should have been cleaned up a long time ago. I'm also happy to do it myself, let me know! Kind regards, Fokko Op wo 14 aug 2024 om 17:49 schreef Xuanwo : > Hi, > > The dev branch of SVN is used to host artifacts awaiting a vote. It > increases the

[VOTE] Release Apache Iceberg Rust 0.3.0 RC1

Hello, Apache Iceberg Rust Community, This is a call for a vote to release Apache Iceberg rust version 0.3.0. The tag to be voted on is 0.3.0. The release candidate: https://dist.apache.org/repos/dist/dev/iceberg/iceberg-rust-0.3.0-rc.1/ Keys to verify the release candidate: https://downloads

Re: [DISCUSS] Cleanup svn dev/iceberg

Got it. I will clean them up. On Wed, Aug 14, 2024, at 23:54, Fokko Driesprong wrote: > Hey Xuanwo, > > Feel free to clean those up as they should have been cleaned up a long time > ago. I'm also happy to do it myself, let me know! > > Kind regards, > Fokko > > Op wo 14 aug 2024 om 17:49 schre

Re: [DISCUSS] Cleanup svn dev/iceberg

Thanks! Kind regards, Fokko Op wo 14 aug 2024 om 17:57 schreef Xuanwo : > Got it. I will clean them up. > > On Wed, Aug 14, 2024, at 23:54, Fokko Driesprong wrote: > > Hey Xuanwo, > > Feel free to clean those up as they should have been cleaned up a long > time ago. I'm also happy to do it mysel

Re: [DISCUSS] REST Endpoint discovery

Hey Dmitri, this proposal is the result of the community feedback from the Capabilities proposal. Ultimately the capabilities turned out to entail more complexity than necessary and so this proposal solves the core problem while keeping complexity and spec changes to an absolute minimum. Eduard

Re: [DISCUSS] Changing namespace separator in REST spec

I do object the plan to make that separation character configurable, because there is no need to do this (see below), so there's no need for any code change. Instead having a proper, mostly human readable escaping mechanism, which is possible, should be implemented with Iceberg REST v2. Here's

Re: [VOTE] Release Apache PyIceberg 0.7.1rc2

- validated signatures and checksums - checked license - ran tests and test-coverage with Python 3.9.12 +1 (non-binding) André Anastácio On Tuesday, August 13th, 2024 at 10:19 PM, Sung Yun wrote: > Hi Everyone, > > I propose that we release the following RC as the official PyIceberg 0.7.1 >

Re: [VOTE] Release Apache PyIceberg 0.7.1rc2

+1 (binding) Thanks Sung for running this 🙌 - Validated signatures/checksums/license - Ran some basic tests (3.10) Kind regards, Fokko Op wo 14 aug 2024 om 19:57 schreef André Luis Anastácio : > >- validated signatures and checksums > > >- checked license > > >- ran tests and test-

Re: [DISCUSS] Materialized Views: Lineage and State information

I'd like to hear Jan's feedback on using UUID and normalizing the view lineage. I'm on board with this change. I updated the fully spec'd out example using UUID and a normalized view linage: https://docs.google.com/document/d/1UnhldHhe3Grz8JBngwXPA6ZZord1xMedY5ukEhZYF-A/edit#heading=h.o6yn2lnpxo

Re: [DISCUSS] Variant Spec Location

I'm really excited about the introduction of variant type to Iceberg, but I want to raise concerns about forking the spec. I feel like preemptively forking would create the situation where we end up diverging because there's little reason to work with both communities to evolve in a way that benef

Re: [DISCUSS] Variant Spec Location

I'm not really in favor of linking and annotating as that just makes things more complicated and still is essentially forking just with more steps. If we just track our annotations / modifications to a single commit/version then we have the same issue again but now you have to go to multiple sourc

Re: [DISCUSS] Materialized Views: Lineage and State information

Thanks Benny. For refs, I am +1 to represent them as UUID + optional ref, although we can iterate ohe exact JSON structure (e.g., another option is splitting for (UUID) state from (UUID + ref) state into two separate higher-level fields). Generally agree on REFRESH VIEW strategy could be up to the

Re: [DISCUSS] Cleanup svn dev/iceberg

Thanks for taking care of this! And good to raise it on dev@ since we should be cleaning up old RCs for all the implementations. On Wed, Aug 14, 2024 at 8:58 AM Fokko Driesprong wrote: > Thanks! > > Kind regards, > Fokko > > Op wo 14 aug 2024 om 17:57 schreef Xuanwo : > >> Got it. I will clean t

Re: [Early Feedback] Variant and Subcolumnarization Support

Hi Nick, Your understanding is correct. The null in the Variant spec is meant to encode a JSON null. A row-level value can be SQL null as in any nullable column, but within a Variant value, there is only the Variant-encoded null (i.e. JSON null). Some of the Spark expressions (e.g. cast to a non-V

Re: [Early Feedback] Variant and Subcolumnarization Support

Hi David, just to clarify, I think we can shred arrays with json nulls without having to use untyped_value column, is this correct? Selcuk On Wed, Aug 14, 2024 at 11:31 PM David Cashman wrote: > Hi Nick, > > Your understanding is correct. The null in the Variant spec is meant > to encode a JSON

Re: [Early Feedback] Variant and Subcolumnarization Support

Hi Selcuk, that's a good point. I don't think the spec doesn't discuss how a null in an array should be interpreted. I had assumed that it would be an invalid state (and probably should have said so explicitly), but I agree that we could specify that it should be interpreted as a JSON null. Thanks

Re: [DISCUSS] Variant Spec Location

+1 for copying the spec into our repository, I think we need to own it fully as a part of the table spec, and we can build compatibility through tests. -Jack On Wed, Aug 14, 2024 at 12:52 PM Russell Spitzer wrote: > I'm not really in favor of linking and annotating as that just makes > things m

Re: [VOTE] Merge REST spec clarification on how servers should handle unknown updates/requirements

+1 -Jack On Wed, Aug 14, 2024 at 8:39 AM Daniel Weeks wrote: > +1 > > On Tue, Aug 13, 2024 at 7:34 PM xianjin wrote: > >> +1 >> >> On Aug 14, 2024, at 2:24 AM, Ryan Blue >> wrote: >> >>  >> +1 >> >> On Tue, Aug 13, 2024 at 8:59 AM Yufei Gu wrote: >> >>> +1 >>> Yufei >>> >>> >>> On Tue, Aug

Re: Welcome Péter, Amogh and Eduard to the Apache Iceberg PMC

Congratulations! On Wed, Aug 14, 2024 at 8:12 AM Amogh Jahagirdar <2am...@gmail.com> wrote: > Thanks all, and congrats to Péter and Eduard! > > On Wed, Aug 14, 2024 at 6:40 AM Steve Loughran > wrote: > >> >> congratulations all. >> >> On Tue, 13 Aug 2024 at 21:25, Russell Spitzer >> wrote: >> >

Re: [VOTE] Release Apache PyIceberg 0.7.1rc2

+1 (non-binding) Verified signatures/checksums/licenses. Ran unit and integration tests. On Thu, Aug 15, 2024 at 2:42 AM Fokko Driesprong wrote: > +1 (binding) > > Thanks Sung for running this 🙌 > > - Validated signatures/checksums/license > - Ran some basic tests (3.10) > > Kind regards, > Fokk

Re: [DISCUSS] Variant Spec Location

Sorry for chiming in late. >From the discussion in https://lists.apache.org/thread/xcyytoypgplfr74klg1z2rgjo6k5b0sq, I don't quite understand why it is logistically complicated to create a sub-project to hold the variant spec and impl. IMHO, coping the variant type spec into Apache Iceberg has so

Re: Welcome Péter, Amogh and Eduard to the Apache Iceberg PMC

Congrats ! On Thu, Aug 15, 2024 at 10:12 AM Jack Ye wrote: > Congratulations! > > On Wed, Aug 14, 2024 at 8:12 AM Amogh Jahagirdar <2am...@gmail.com> wrote: > >> Thanks all, and congrats to Péter and Eduard! >> >> On Wed, Aug 14, 2024 at 6:40 AM Steve Loughran >> wrote: >> >>> >>> congratulatio

Re: [VOTE] Merge REST spec clarification on how servers should handle unknown updates/requirements

+1 On Thu, Aug 15, 2024 at 10:10 AM Jack Ye wrote: > +1 > > -Jack > > On Wed, Aug 14, 2024 at 8:39 AM Daniel Weeks wrote: > >> +1 >> >> On Tue, Aug 13, 2024 at 7:34 PM xianjin wrote: >> >>> +1 >>> >>> On Aug 14, 2024, at 2:24 AM, Ryan Blue >>> wrote: >>> >>>  >>> +1 >>> >>> On Tue, Aug 13, 2

Re: [DISCUSS] Variant Spec Location

I’m on board with copying the spec into our repository. However, as we’ve talked about, it’s not just a straightforward copy—there are already some divergences. Some of them are under discussion. Iceberg is definitely the best place for these specs. Engines like Trino and Flink can then rely on the