> Would this property cause streaming writes using equality deletes to fail until the table is updated? I’m open to this solution since I think people should definitely be aware of the trade-offs they’re making in their tables.
I don't think we can do such a check on the Iceberg side. As discussed there could be perfectly valid reasons to write equality deletes and keep row lineage. Even positional deletes written by an outdated engine could mess up the lineage information. I would push the responsibility to the engine to check the property when writing updates to a V3 table. If the property is set then queries which would write delete-and-insert type updates should be rejected. Thanks, Peter Amogh Jahagirdar <2am...@gmail.com> ezt írta (időpont: 2025. márc. 21., P, 19:08): > I support enabling row lineage by default primarily because of the > ecosystem benefit that enables engines to rely on lineage without requiring > users to opt in explicitly. This should generally apply to most engines and > integrations. > > > However, as we know there are specific cases in the ecosystem—such as > streaming engines producing equality deletes where we know row lineage > cannot be preserved due to the expensive read and state management that > would be involved. > > > So that said, I think I agree with Peter that having a table property to > indicate whether row lineage is accurate or not would be beneficial. I > think this is preferable to expecting users to understand the nuances of > different engines regarding lineage preservation. It provides users with a > clear indication of what is happening in their table. > > > Thanks, > > > Amogh Jahagirdar >
