Okay, it sounds like we have consensus that it's a good idea to make row lineage required in v3 and that it's a good idea to signal to engines when they can write delete-and-insert changes. I think we need a bit more discussion on how to signal to engines, but in the meantime we can move forward with the row lineage changes. Thanks for the discussion, and I'll come up with a proposal for the property that Peter is suggesting.
Ryan On Mon, Mar 24, 2025 at 7:02 AM Péter Váry <peter.vary.apa...@gmail.com> wrote: > > Would this property cause streaming writes using equality deletes to > fail until the table is updated? I’m open to this solution since I think > people should definitely be aware of the trade-offs they’re making in their > tables. > > I don't think we can do such a check on the Iceberg side. As discussed > there could be perfectly valid reasons to write equality deletes and keep > row lineage. Even positional deletes written by an outdated engine could > mess up the lineage information. > > I would push the responsibility to the engine to check the property when > writing updates to a V3 table. If the property is set then queries which > would write delete-and-insert type updates should be rejected. > > Thanks, > Peter > > Amogh Jahagirdar <2am...@gmail.com> ezt írta (időpont: 2025. márc. 21., > P, 19:08): > >> I support enabling row lineage by default primarily because of the >> ecosystem benefit that enables engines to rely on lineage without requiring >> users to opt in explicitly. This should generally apply to most engines and >> integrations. >> >> >> However, as we know there are specific cases in the ecosystem—such as >> streaming engines producing equality deletes where we know row lineage >> cannot be preserved due to the expensive read and state management that >> would be involved. >> >> >> So that said, I think I agree with Peter that having a table property to >> indicate whether row lineage is accurate or not would be beneficial. I >> think this is preferable to expecting users to understand the nuances of >> different engines regarding lineage preservation. It provides users with a >> clear indication of what is happening in their table. >> >> >> Thanks, >> >> >> Amogh Jahagirdar >> >
