In Flink there are 2 types of CDC streams: - Upsert stream - in this case the sink receives only -D (delete), +I (insert) records - In this case we can't differentiate - Retract stream - in this case the sink receives -D (delete), +I (insert), -U (removed by update), +U (added by update) records - In this case we know if a record is updated, so we can assume that there were a previous version for the record
I'm not sure about Kafka though Steven Wu <stevenz...@gmail.com> ezt írta (időpont: 2025. febr. 12., Sze, 15:42): > > If we want to keep the equality deletes, we might add a marker to the > updated row (for example rowId=-1) that this is an update. The reader and > the compaction could calculate the correct value. > > Peter, this is probably not going to work with equality deletes. The > writer doesn't really know if it is an insert or update, since it didn't > really check if existing data files have the same row key (equality > fields). Otherwise, writers can produce position deletes already. Equality > deletes are cheap for writers, because there are no need to do any > check/join/correlation at all. Equality deletes just blindly claim deletion > of any previous rows ( *if there are any*) marked by the equality key. > > > On Tue, Feb 11, 2025 at 10:11 PM Péter Váry <peter.vary.apa...@gmail.com> > wrote: > >> Hi Russell, >> >> Thanks for bringing this up! >> I think equality deletes are not the root of the problem here. >> >> - If we have a positional delete, and the new row doesn't include the old >> rowId, then the lineage info is lost. >> - If we have an equality delete, and the new row contains the rowId, then >> we have the lineage info >> >> That said, it is unlikely to have rowId for the new rows for writers >> (Flink, Kafka Connect) who are currently using equality deletes for >> updates, because of the exact reason they're using equality deletes (has to >> defer the lookup for later). Maybe we can add checks for these engines to >> warn when they try to write a table which has row lineage enabled. >> >> If we want to keep the equality deletes, we might add a marker to the >> updated row (for example rowId=-1) that this is an update. The reader and >> the compaction could calculate the correct value. That said, I think it >> would be better to invest in finding a replacement for equality deletes >> than investing more in it. Maybe adding Iceberg indexes which would allow >> fast lookup and conversion at commit time, or push those indexes to the >> file format level? >> >> Thanks, Peter >> >> >> On Wed, Feb 12, 2025, 06:38 Gang Wu <ust...@gmail.com> wrote: >> >>> Thanks Steven for the explanation! Yes, you're right that solely >>> rewriting delete files does not help. >>> >>> IIUC, Iceberg is the only table format that does not produce changelog >>> files. Is there any chance to recompute the row_id of updated rows by >>> tracking changes of the identifier fields between snapshots during the >>> rewrite and produce changelog files? >>> >>> I'm not asking to add changelogs support since it is a large design >>> choice. Just want to brainstorm it. >>> >>> On Wed, Feb 12, 2025 at 11:55 AM Steven Wu <stevenz...@gmail.com> wrote: >>> >>>> I am fine with the proposed spec change. While it "supports/allows" >>>> equality deletes, row lineage semantics needn't/can't be maintained >>>> properly for equality deletes (compared to position deletes). Gang pointed >>>> out a couple issues with the implications. But we have no choice but to >>>> live with those implications due to how equality deletes behave. >>>> >>>> Gang, rewriting equality deletes to position deletes doesn't really >>>> help in this case. To have correct lineage, the row update is supposed to >>>> have the row_id carried over from the previous row (equality deleted row) >>>> during the write phase with equality deletes. Instead, this spec change now >>>> says the updated row is a complete new row with new row_id. >>>> >>>> On Tue, Feb 11, 2025 at 7:39 PM Gang Wu <ust...@gmail.com> wrote: >>>> >>>>> Hi Russell, >>>>> >>>>> Thanks for supporting equality deletes to row lineage! >>>>> >>>>> > accept that "updates" will be treated as "delete" and "insert" >>>>> >>>>> I would say that it has obvious drawbacks below (though it is better >>>>> than not supported): >>>>> 1) updates will be populated differently when outputting changelogs to >>>>> users or downstream databases >>>>> 2) lead to more computation for incremental processing like refreshing >>>>> materialized views >>>>> >>>>> At the same time, I would like to ask if it would help if we support >>>>> rewriting equality deletes to position deletes. >>>>> There was an effort but it has been closed: >>>>> https://github.com/apache/iceberg/pull/2216 >>>>> >>>>> Best, >>>>> Gang >>>>> >>>>> >>>>> On Wed, Feb 12, 2025 at 7:25 AM Russell Spitzer < >>>>> russell.spit...@gmail.com> wrote: >>>>> >>>>>> Hi Y'all, >>>>>> >>>>>> As we have been working on the row lineage implementation I've been >>>>>> reached out to by a few folks >>>>>> in the community who are interested in changing our defined behavior >>>>>> around equality deletes. >>>>>> >>>>>> Currently when Row Lineage is enabled, the spec says to disable >>>>>> equality deletes for the table. >>>>>> >>>>>> In the interest of compatibility with Flink and other Equality delete >>>>>> producers, I originally wrote >>>>>> that we would simply treat all equality delete based updates as a >>>>>> pure insert and >>>>>> delete. At the time, some folks thought this was too open and worried >>>>>> that it would be poor behavior which >>>>>> led to the current restriction. >>>>>> >>>>>> Now that we are actually implementing I think there have been some >>>>>> changes of heart and that we >>>>>> should go back to the original design. I'd like to see if we have >>>>>> consensus >>>>>> in the community to change the wording back and allow equality >>>>>> deletes. >>>>>> >>>>>> PR: https://github.com/apache/iceberg/pull/12230 >>>>>> >>>>>> The TLDR; >>>>>> >>>>>> Allow equality deletes with row lineage but accept that "updates" >>>>>> will be treated as "delete" and "insert" >>>>>> >>>>>> Thanks for your time, >>>>>> Russ >>>>>> >>>>>
