Re: [Discussion] Spec change for Row Lineage - Allow Equality Deletes

In Flink there are 2 types of CDC streams: - Upsert stream - in this case the sink receives only -D (delete), +I (insert) records - In this case we can't differentiate - Retract stream - in this case the sink receives -D (delete), +I (insert), -U (removed by update), +U (added by updat

Re: [Discussion] Spec change for Row Lineage - Allow Equality Deletes

> If we want to keep the equality deletes, we might add a marker to the updated row (for example rowId=-1) that this is an update. The reader and the compaction could calculate the correct value. Peter, this is probably not going to work with equality deletes. The writer doesn't really know if it

Re: [VOTE] Release Apache Iceberg 1.8.0 RC0

+1 (binding) - all sigs/checksum/RAT checks - build and test with JDK17 - did some tests with Spark/DVs locally Thanks, Amogh Jahagirdar On Wed, Feb 12, 2025 at 7:36 AM Jean-Baptiste Onofré wrote: > +1 (non binding) > > I checked: > - hash and checksum are good > - all LICENSE and NOTICE are g

Re: [VOTE] Release Apache Iceberg 1.8.0 RC0

Thanks everyone who participated in the vote for Release Apache Iceberg 1.8.0 RC0. The vote result is: +1: 6 (binding), 6 (non-binding) +0: 0 (binding), 0 (non-binding) -1: 0 (binding), 0 (non-binding) Therefore, the release candidate is passed. On Thu, Feb 13, 2025 at 5:41 AM huaxin gao wrote

Re: Table metadata swap not work for REST Catalog (#12134)

Agree. Gravitino‘s Iceberg REST service provides similar mechanism to notify the changes. You can see https://gravitino.apache.org/docs/0.8.0-incubating/iceberg-rest-service#event-listener Yufei Gu 于2025年2月11日周二 10:01写道： > The push-based mirroring highlighted by Ryan is a popular use case. > Pol

Re: [VOTE] Release Apache Iceberg 1.8.0 RC0

+1 (non-binding) - Checked signature/SHA512 - Ran RAT check - Ran tests on JDK17 Thanks, Steve Zhang > On Feb 12, 2025, at 9:52 AM, Eduard Tudenhöfner > wrote: > > +1 (binding) > > Verified sigs/checksums/build/tests with JDK17 > > I also saw the same TestS3FileIO failures that Kevin repo

Re: [Discussion] Spec change for Row Lineage - Allow Equality Deletes

Let's say, that rowId -1 means that the rowId needs to be calculated based on the primary key. In this case a maintenance job could query the previous snapshot of the table and find the previous version of the row with the given primary key, and update the row (and the newer versions of the row) to

Re: [Discussion] Spec change for Row Lineage - Allow Equality Deletes

That would be very difficult since we can't actually tell what the primary key is at any given point in time. We also would then have to define what to do if multiple records with the same key exists etc .. I think it's better to just leave that as an exercise to any engine that really wants to imp

Re: [VOTE] Release Apache Iceberg 1.8.0 RC0

+1 (binding) Verified sigs/sums/license/build/test (Java 17) I also manually tested a number of cases with format v3 and DVs. -Dan On Wed, Feb 12, 2025 at 8:57 AM Ajantha Bhat wrote: > +1 (non-binding) > > * validated checksum and signature > * checked license docs & ran RAT checks > * ran bu

Re: [VOTE] Release Apache Iceberg 1.8.0 RC0

+1 (binding) Verified sigs/checksums/build/tests with JDK17 I also saw the same *TestS3FileIO* failures that Kevin reported earlier (which passed on a subsequent run) and I do agree with Ryan that we may want to move Docker-specific tests to the *integrationTest* configuration. I opened #12236

Re: [Discussion] Spec change for Row Lineage - Allow Equality Deletes

I'm not sure I follow how one could figure out the equality delete row ID after the fact. Won't I need to use some other primary key identifier and do a shuffle join to line it up with existing records? On Wed, Feb 12, 2025 at 8:57 AM Péter Váry wrote: > In Flink there are 2 types of CDC streams

Re: [VOTE] Release Apache Iceberg 1.8.0 RC0

+1 (non-binding) Verified signatures, checksums, build and ran some tests on my local. Thanks Amogh for driving the release! Best, Huaxin On Wed, Feb 12, 2025 at 11:28 AM Honah J. wrote: > +1 (binding) > > Verified signatures, checksum, build, and tests. > > Also noticed the same set of flaky

Re: [VOTE] Add RemoveSchemas update type to REST spec

+1 On Wed, Feb 12, 2025 at 1:40 PM Gang Wu wrote: > +1 (non-binding) > > On Wed, Feb 12, 2025 at 6:17 AM Amogh Jahagirdar <2am...@gmail.com> wrote: > >> +1 thanks for driving this Gabor! >> >> On Wed, Feb 12, 2025 at 2:35 AM rdb...@gmail.com >> wrote: >> >>> +1 >>> >>> On Tue, Feb 11, 2025 at 1

Re: [VOTE] Release Apache Iceberg 1.8.0 RC0

+1 (binding) Verified signatures, checksum, build, and tests. Also noticed the same set of flaky tests that Kevin reported. Thanks Amogh for driving this! Best regards, Honah On Wed, Feb 12, 2025 at 9:52 AM Eduard Tudenhöfner wrote: > +1 (binding) > > Verified sigs/checksums/build/tests wit

Re: [VOTE] Release Apache Iceberg 1.8.0 RC0

+1 (non-binding) * validated checksum and signature * checked license docs & ran RAT checks * ran build and tests with JDK11 - Ajantha On Wed, Feb 12, 2025 at 10:37 AM Fokko Driesprong wrote: > +1 (binding) > > Checked signatures/tests/licenses and plugged 1.8.0 into the CI of > PyIceberg

Re: [VOTE] Release Apache Iceberg 1.8.0 RC0

+1 (binding) Checked signatures/tests/licenses and plugged 1.8.0 into the CI of PyIceberg . Thanks for running this Amogh! Kind regards, Fokko Op wo 12 feb 2025 om 16:57 schreef Amogh Jahagirdar <2am...@gmail.com>: > +1 (binding) > > - all si

[VOTE] Add overwriteRequested to RegisterTableRequest in REST spec

Hi Iceberg Community, I'm working on supporting the registration of iceberg metadata for an existing table in the catalog. As part of this work, I'm proposing to add an optional boolean field in RegisterTableRequest. I'd like to start a vote on this REST spec change: https://github.com/apac