Hi Prashant, Interesting stuff!
I have a few questions: 1. I understand that it is easier to apply positional deletes than equality deletes. What about the conversion cost? Based on your experience, what is the cost difference between a conversion and a full rewrite? When does it worth to do a delete conversion, and when does it worth to do a full data rewrite? 2. If writers still write equality deletes, how would that work with the row lineage? We still would need to solve the problem for the rows updated by the "not yet converted" deletes. Do you have a proposal for this? We might want to add that to the linked document. 3. If/when we move forward with this conversion, it would be nice to choose an implementation which would allow the the Flink table maintenance to reuse as much as possible. Thanks, Peter On Sat, Sep 14, 2024, 02:08 Szehon Ho <szehon.apa...@gmail.com> wrote: > +1, Id be happy to see this feature. > > Thanks > Szehon > > On Fri, Sep 13, 2024 at 10:33 AM Prashant Singh <prashant010...@gmail.com> > wrote: > >> Hi All, >> >> Starting this thread to revive the discussion on converting Equality >> Deletes as Position deletes and see if this is something community wants >> now (Happy to contribute in this) considering : >> 1/ Now it's not just flink but other writers such Kafka-Connect [1] >> debezium server for iceberg [2] have emerged which emit equality deletes >> and equality deletes doesn't goes well with the readers in general. >> 2/ We have some features like Row lineage [3] on which we have taken >> decided we will not support it with equality deletes >> 3/ We are doing considerable enhancements in Position deletes in v3 [4] >> >> I see some past work has been done in this context [5] and we have an >> action already in our repo. >> >> Let me know your thoughts. >> >> Regards, >> Prashant >> >> [1] https://github.com/tabular-io/iceberg-kafka-connect/tree/main >> [2] https://github.com/memiiso/debezium-server-iceberg >> [3] >> https://docs.google.com/document/d/146YuAnU17prnIhyuvbCtCtVSavyd5N7hKryyVRaFDTE/edit#heading=h.f2e8ffw3fu7n >> [4] >> https://docs.google.com/document/d/18Bqhr-vnzFfQk1S4AgRISkA_5_m5m32Nnc2Cw0zn2XM/edit >> [5] >> https://github.com/apache/iceberg/blob/main/api/src/main/java/org/apache/iceberg/actions/ConvertEqualityDeleteFiles.java >> >
