Re: [ANNOUNCE] Release Apache Iceberg Rust v0.3.0

Thanks Xuanwo! This is exciting and a big milestone, thanks everyone for working on this! Kind regards, Fokko Op wo 21 aug 2024 om 07:49 schreef Renjie Liu : > Thanks Xuanwo for driving this release! Thanks to all contributors! > > On Wed, Aug 21, 2024 at 1:06 PM Xuanwo wrote: > >> Hi all, >> >

Re: [ANNOUNCE] Release Apache Iceberg Rust v0.3.0

Thanks Xuanwo for driving this release! Thanks to all contributors! On Wed, Aug 21, 2024 at 1:06 PM Xuanwo wrote: > Hi all, > > The Apache Iceberg Rust community is pleased to announce > that Apache Iceberg Rust v0.3.0 has been released! > > Iceberg is an open table format for analytic datasets,

Re: Type promotion in v3

Thanks Dan, I'm pretty strongly opposed to the idea of assigning new field ids as part > of type promotion. I understand what we're trying to accomplish, but I > just don't think that's the right mechanism to achieve it. The field id > specifically identifies the field and shouldn't change as >

[ANNOUNCE] Release Apache Iceberg Rust v0.3.0

Hi all, The Apache Iceberg Rust community is pleased to announce that Apache Iceberg Rust v0.3.0 has been released! Iceberg is an open table format for analytic datasets, and Iceberg Rust is the native Rust implementation of Iceberg. The notable changes since v0.3.0 include: - Added Sync + Sen

clarification on changelog behavior for equality deletes

Hi, I have a PR open to add changelog support for the case where delete files are present (https://github.com/apache/iceberg/pull/10935). I have a question about what the changelog should emit in the following scenario: The table has a schema with a primary key/identifier column PK and additional

Re: [DISCUSS] Release source and binary verification

Hi, > I would add an additional check on the source distribution: the source > distribution should not contain any unexpected binary file (gradle > wrapper is OK, but other binary should be avoided). It’s not OK to include the gradle wrapper in the source release. A source release can't include

Re: [VOTE] REST Endpoint discovery

+1 (non-binding) Thank you Eduard! This is a great feature enhancement to the catalog. Thumbs up! On Tue, Aug 20, 2024 at 5:27 PM Amogh Jahagirdar <2am...@gmail.com> wrote: > +1 > Thanks for driving this Eduard! > > On Tue, Aug 20, 2024 at 3:08 PM Ryan Blue > wrote: > >> +1 >> >> On Tue, Aug 20

Re: [VOTE] REST Endpoint discovery

+1 Thanks for driving this Eduard! On Tue, Aug 20, 2024 at 3:08 PM Ryan Blue wrote: > +1 > > On Tue, Aug 20, 2024 at 1:46 PM Christian Thiel > wrote: > >> + 1 (non-binding) >> >> > > -- > Ryan Blue > Databricks >

Re: [VOTE] REST Endpoint discovery

+1 On Tue, Aug 20, 2024 at 1:46 PM Christian Thiel wrote: > + 1 (non-binding) > > -- Ryan Blue Databricks

Re: [Discuss] Geospatial Support

Hi all Please take a look at the proposed spec change to support Geo type for V3 in : https://github.com/apache/iceberg/pull/10981, and comment or otherwise let me know your thoughts. Just as an FYI it incorporated the feedback from our last meeting (with Snowflake and Wherobots engineers). Than

Re: [VOTE] REST Endpoint discovery

+ 1 (non-binding)

Re: [VOTE] REST Endpoint discovery

+1 On Tue, Aug 20, 2024 at 12:37 PM Russell Spitzer wrote: > +1 > > On Tue, Aug 20, 2024 at 2:32 PM Walaa Eldin Moustafa < > wa.moust...@gmail.com> wrote: > >> +1 non-biding >> >> Thanks for driving this Eduard. >> >> On Tue, Aug 20, 2024 at 12:17 PM Daniel Weeks wrote: >> >>> +1 >>> >>> On Tue

Re: Type promotion in v3

I'm pretty strongly opposed to the idea of assigning new field ids as part of type promotion. I understand what we're trying to accomplish, but I just don't think that's the right mechanism to achieve it. The field id specifically identifies the field and shouldn't change as attributes change (na

Re: [VOTE] REST Endpoint discovery

+1 On Tue, Aug 20, 2024 at 2:32 PM Walaa Eldin Moustafa wrote: > +1 non-biding > > Thanks for driving this Eduard. > > On Tue, Aug 20, 2024 at 12:17 PM Daniel Weeks wrote: > >> +1 >> >> On Tue, Aug 20, 2024 at 11:19 AM Yufei Gu wrote: >> >>> +1 >>> >>> Yufei >>> >>> >>> On Tue, Aug 20, 2024 at

Re: [VOTE] REST Endpoint discovery

+1 non-biding Thanks for driving this Eduard. On Tue, Aug 20, 2024 at 12:17 PM Daniel Weeks wrote: > +1 > > On Tue, Aug 20, 2024 at 11:19 AM Yufei Gu wrote: > >> +1 >> >> Yufei >> >> >> On Tue, Aug 20, 2024 at 11:16 AM Eduard Tudenhöfner < >> etudenhoef...@apache.org> wrote: >> >>> Hey everyon

Re: [VOTE] REST Endpoint discovery

+1 On Tue, Aug 20, 2024 at 11:19 AM Yufei Gu wrote: > +1 > > Yufei > > > On Tue, Aug 20, 2024 at 11:16 AM Eduard Tudenhöfner < > etudenhoef...@apache.org> wrote: > >> Hey everyone, >> >> I'd like to vote on PR #10928 >> which adds a way for REST >>

Re: [VOTE] REST Endpoint discovery

+1 Yufei On Tue, Aug 20, 2024 at 11:16 AM Eduard Tudenhöfner < etudenhoef...@apache.org> wrote: > Hey everyone, > > I'd like to vote on PR #10928 > which adds a way for REST > servers to communicate to clients what endpoints it supports via a new >

[VOTE] REST Endpoint discovery

Hey everyone, I'd like to vote on PR #10928 which adds a way for REST servers to communicate to clients what endpoints it supports via a new *endpoints* field in the *CatalogConfig* of the *v1/config* endpoint. Discuss thread can be found here: https

Re: [DISCUSS] REST Endpoint discovery

Thanks Yufei and everyone else that participated in the discussion. I'll open a vote shortly to get this moving. On Tue, Aug 20, 2024 at 8:04 PM Yufei Gu wrote: > I'm fine with the approach to present the endpoint ID as a plain string. > For any extension in the future, we can add a new field pa

Re: [DISCUSS] REST Endpoint discovery

I'm fine with the approach to present the endpoint ID as a plain string. For any extension in the future, we can add a new field parallel with the endpoint array. Thanks a lot for driving this, Edward. Yufei On Tue, Aug 20, 2024 at 8:05 AM Eduard Tudenhöfner wrote: > @Karuppayya: I've answere

Re: Type promotion in v3

Hi Fokko, > In this case, we still need to keep the schemas. As an example: The example you gave is close to what I was imagining (if we get to details I might have a slightly different organization). This might be semantic, but I don't see this as keeping "schemas", since all data is present i

Re: [VOTE] Release Apache Iceberg 1.6.1 RC1

Hi -1 (non-binding) I verified source tarball matches the git tag (except it lacks jitpack.yml, docs/ and 'examples/Convert table to Iceberg.ipynb'). However, i noted that source tarball verification is not part of https://iceberg.apache.org/how-to-release/#validating-a-source-release-candidate .

Re: [DISCUSS] Materialized Views: Lineage and State information

Theoretically, we could have multiple catalogs each with different table name entries but referring to the same Iceberg table metadata, and hence same UUIDs (view metadata cannot be shared since they are strongly bound to the catalog identifiers). I understand this is not an everyday scenario but i

Re: Community sync

Copied from Calendar Invite https://www.google.com/url?q=https://meet.google.com/ujy-njjo-vre&sa=D&source=calendar&ust=1724601702621697&usg=AOvVaw1Rd0-RlNwoXE-OIDTVtAGC Triweekly Iceberg meeting for anyone wanting to get involved in the Iceberg development, documentation, or hear about the ro

Community sync

Based on previous emails in this list I got the impression that a community sync is coming up, possibly tomorrow morning. Where can I fing the meeting information so that I may listen in? -Steve Lessard, Teradata

Re: [DISCUSS] Release source and binary verification

Hi Piotr It sounds reasonable to me. If you mean reproducible build (the build from the same source should create the same artifact), I submitted some changes a while ago, adding fixed file order, etc. We should be good (https://github.com/apache/iceberg/pull/8826). I would add an additional che

Re: [DISCUSS] Release source and binary verification

I think these are reasonable to add, we probably should also verify there are no binaries of any kind in the release tarball. Sometimes builds accidentally leak these. On Tue, Aug 20, 2024 at 8:36 AM Piotr Findeisen wrote: > Hi All, > > Hi > > The release verification [1] includes testing releas

Re: [DISCUSS] REST Endpoint discovery

@Karuppayya: I've answered your question in the doc. @Yufei: Part of the reason why the proposed format is so simple is because there were different opinions about what level of granularity we should have in the context of REST capabilities, which then led to this version. It seems people are gene

Re: Table schema and partition spec update

Hi Péter, > I have seen requirements for accommodating partitioning scheme changes when > the Table has been changed. This is similar with request I received from users. It’s possible to update/refresh the table spec/schema in the next checkpoint without Flink Job restart. It requires some ex

Re: Type promotion in v3

Hi Micah, > I think the idea with Parquet files is one would no longer use a map to track > these statistics but instead have a column per field-id/statistics pair. > …. > This is similar to how partition values are stored today in Avro. And I > don’t think there is anything stopping from doi

Re: [VOTE] Release Apache Iceberg 1.6.1 RC1

> TestHadoopCommits > testConcurrentFastAppends(File) FAILED I have seen this test a bit flaky before On Tue, Aug 20, 2024 at 6:31 AM Jean-Baptiste Onofré wrote: > +1 (non binding) > > I checked: > - download links are OK (both on dist and Maven Staging repo) > - build passed on the tag using J

Re: [VOTE] Spec changes in preparation for v3

+1 On Tue, Aug 20, 2024 at 4:16 AM xianjin wrote: > +1 (non-binding) > Sent from my iPhone > > On Aug 20, 2024, at 7:56 AM, Manu Zhang wrote: > >  > +1 (non-binding) > > Micah Kornfield 于2024年8月20日 周二07:44写道： > >> +1 (non-binding) >> >> On Mon, Aug 19, 2024 at 4:33 PM Steve Zhang >> wrote: >>

[DISCUSS] Release source and binary verification

Hi All, Hi The release verification [1] includes testing release source tarball builds and also testing the binaries with downstream projects. Does it also contain, should it contain or is it a conscious omission of: 1. verifying the source tarball is what it should be (source matches the git r

Re: [VOTE] Release Apache Iceberg 1.6.1 RC1

+1 (non binding) I checked: - download links are OK (both on dist and Maven Staging repo) - build passed on the tag using JDK11, including the tests (I'm not able to reproduce Renjie's issue) - checksum and signature are good - ASF header present in expected files - no unexpected binary files foun

Re: Type promotion in v3

> > Yes, I was thinking it would be a recursive structure that tracked each > change. Cleanup could be achieved by also tracking schema ID of the last > time the field was present along with the schema ID of the written data > files in manifests (as discussed on the other threads), and cleaning up

Re: [DISCUSS] Adding RemovePartitionSpecsUpdate update type to REST

+1 Thanks for working on this Op di 20 aug 2024 om 04:16 schreef xianjin : > +1 from my side as well. > > Sent from my iPhone > > On Aug 20, 2024, at 9:09 AM, Yufei Gu wrote: > >  > > +1, the new spec looks good to me. It seems like the client-side handling > the heavy lifting of figuring out w

Re: [DISCUSS] Iceberg 1.6.1 release

Hi Sorry to have been quiet: I was on vacation during the past two weeks (with limited connection while hiking). I saw that your started the 1.6.1 release. Thanks ! I will check the release. Regarding Avro, I will move forward on 1.12.0 release. Regards JB Le mer. 7 août 2024 à 16:14, Piotr Fi

Re: [VOTE] Make namespace separator configurable in REST Spec

-1 (nb) On 16.08.24 17:46, Dmitri Bourlatchkov wrote: +1 (nb) to the spec change. Cheers, Dmitri. On Fri, Aug 16, 2024 at 4:31 AM Eduard Tudenhöfner wrote: Hey everyone, as I mentioned on the DISCUSS thread, this is providing a simple path forward for users of the V1 APIs (mak