Hi Ajantha, I'm proposing exploring a view-based approach similar to the changelog-mirror table pattern[1] rather than supporting delta writers for Kafka connect Iceberg sink.
1. https://www.tabular.io/apache-iceberg-cookbook/data-engineering-cdc-table-mirroring/ On Tue, Nov 19, 2024 at 7:38 PM Jean-Baptiste Onofré <j...@nanthrax.net> wrote: > I don’t think it’s a problem while an alternative is explored (the JDK > itself does that pretty often). > So it’s up to the community: of course I’m against removing it without > solid alternative, but deprecation is fine imho. > > Regards > JB > > Le mar. 19 nov. 2024 à 12:19, Ajantha Bhat <ajanthab...@gmail.com> a > écrit : > >> - ok for deprecate equality deletes >>> - not ok to remove it >> >> >> @JB: I don't think it is a good idea to use deprecated functionality in >> the new feature development. >> Hence, my specific question was about kafka connect upsert operation. >> >> @Manu: I meant the delta writers for kafka connect Iceberg sink (which in >> turn used for upsetting the CDC records) >> https://github.com/apache/iceberg/issues/10842 >> >> >> - Ajantha >> >> >> >> On Tue, Nov 19, 2024 at 3:08 PM Manu Zhang <owenzhang1...@gmail.com> >> wrote: >> >>> I second Anton's proposal to standardize on a view-based approach to >>> handle CDC cases. >>> Actually, it's already been explored in detail[1] by Jack before. >>> >>> [1] Improving Change Data Capture Use Case for Apache Iceberg >>> <https://docs.google.com/document/d/1kyyJp4masbd1FrIKUHF1ED_z1hTARL8bNoKCgb7fhSQ/edit?tab=t.0#heading=h.94xnx4qg3bnt> >>> >>> >>> On Tue, Nov 19, 2024 at 4:16 PM Jean-Baptiste Onofré <j...@nanthrax.net> >>> wrote: >>> >>>> My proposal is the following (already expressed): >>>> - ok for deprecate equality deletes >>>> - not ok to remove it >>>> - work on position deletes improvements to address streaming use cases. >>>> I think we should explore different approaches. Personally I think a >>>> possible approach would be to find index way to data files to avoid full >>>> scan to find row position. >>>> >>>> My $0.01 :) >>>> >>>> Regards >>>> JB >>>> >>>> Le mar. 19 nov. 2024 à 07:53, Ajantha Bhat <ajanthab...@gmail.com> a >>>> écrit : >>>> >>>>> Hi, What's the conclusion on this thread? >>>>> >>>>> Users are looking for Upsert (CDC) support for OSS Iceberg >>>>> kafka connect sink. >>>>> We only support appends at the moment. Can we go ahead and implement >>>>> the upserts using equality deletes? >>>>> >>>>> >>>>> - Ajantha >>>>> >>>>> On Sun, Nov 10, 2024 at 11:56 AM Vignesh <vignesh.v...@gmail.com> >>>>> wrote: >>>>> >>>>>> Hi, >>>>>> I am reading about iceberg and am quite new to this. >>>>>> This puffin would be an index from key to data file. Other use cases >>>>>> of Puffin, such as statistics are at a per file level if I understand >>>>>> correctly. >>>>>> >>>>>> Where would the puffin about key->data file be stored? It is a >>>>>> property of the entire table. >>>>>> >>>>>> Thanks, >>>>>> Vignesh. >>>>>> >>>>>> >>>>>> On Sat, Nov 9, 2024 at 2:17 AM Shani Elharrar >>>>>> <sh...@upsolver.com.invalid> wrote: >>>>>> >>>>>>> JB, this is what we do, we write Equality Deletes and periodically >>>>>>> convert them to Positional Deletes. >>>>>>> >>>>>>> We could probably index the keys, maybe partially index using bloom >>>>>>> filters, the best would be to put those bloom filters inside puffin. >>>>>>> >>>>>>> Shani. >>>>>>> >>>>>>> On 9 Nov 2024, at 11:11, Jean-Baptiste Onofré <j...@nanthrax.net> >>>>>>> wrote: >>>>>>> >>>>>>> >>>>>>> Hi, >>>>>>> >>>>>>> I agree with Peter here, and I would say that it would be an issue >>>>>>> for multi-engine support. >>>>>>> >>>>>>> I think, as I already mentioned with others, we should explore an >>>>>>> alternative. >>>>>>> As the main issue is the datafile scan in streaming context, maybe >>>>>>> we could find a way to "index"/correlate for positional deletes with >>>>>>> limited scanning. >>>>>>> I will think again about that :) >>>>>>> >>>>>>> Regards >>>>>>> JB >>>>>>> >>>>>>> On Sat, Nov 9, 2024 at 6:48 AM Péter Váry < >>>>>>> peter.vary.apa...@gmail.com> wrote: >>>>>>> >>>>>>>> Hi Imran, >>>>>>>> >>>>>>>> I don't think it's a good idea to start creating multiple types of >>>>>>>> Iceberg tables. Iceberg's main selling point is compatibility between >>>>>>>> engines. If we don't have readers and writers for all types of tables, >>>>>>>> then >>>>>>>> we remove compatibility from the equation and engine specific formats >>>>>>>> always win. OTOH, if we write readers and writers for all types of >>>>>>>> tables >>>>>>>> then we are back on square one. >>>>>>>> >>>>>>>> Identifier fields are a table schema concept and used in many cases >>>>>>>> during query planning and execution. This is why they are defined as >>>>>>>> part >>>>>>>> of the SQL spec, and this is why Iceberg defines them as well. One use >>>>>>>> case >>>>>>>> is where they can be used to merge deletes (independently of how they >>>>>>>> are >>>>>>>> manifested) and subsequent inserts, into updates. >>>>>>>> >>>>>>>> Flink SQL doesn't allow creating tables with partition transforms, >>>>>>>> so no new table could be created by Flink SQL using transforms, but >>>>>>>> tables >>>>>>>> created by other engines could still be used (both read an write). >>>>>>>> Also you >>>>>>>> can create such tables in Flink using the Java API. >>>>>>>> >>>>>>>> Requiring partition columns be part of the identifier fields is >>>>>>>> coming from the practical consideration, that you want to limit the >>>>>>>> scope >>>>>>>> of the equality deletes as much as possible. Otherwise all of the >>>>>>>> equality >>>>>>>> deletes should be table global, and they should be read by every >>>>>>>> reader. We >>>>>>>> could write those, we just decided that we don't want to allow the >>>>>>>> user to >>>>>>>> do this, as it is most cases a bad idea. >>>>>>>> >>>>>>>> I hope this helps, >>>>>>>> Peter >>>>>>>> >>>>>>>> On Fri, Nov 8, 2024, 22:01 Imran Rashid >>>>>>>> <iras...@cloudera.com.invalid> wrote: >>>>>>>> >>>>>>>>> I'm not down in the weeds at all myself on implementation details, >>>>>>>>> so forgive me if I'm wrong about the details here. >>>>>>>>> >>>>>>>>> I can see all the viewpoints -- both that equality deletes enable >>>>>>>>> some use cases, but also make others far more difficult. What >>>>>>>>> surprised me >>>>>>>>> the most is that Iceberg does not provide a way to distinguish these >>>>>>>>> two >>>>>>>>> table "types". >>>>>>>>> >>>>>>>>> At first, I thought the presence of an identifier-field ( >>>>>>>>> https://iceberg.apache.org/spec/#identifier-field-ids) indicated >>>>>>>>> that the table was a target for equality deletes. But, then it turns >>>>>>>>> out >>>>>>>>> identifier-fields are also useful for changelog views even without >>>>>>>>> equality >>>>>>>>> deletes -- IIUC, they show that a delete + insert should actually be >>>>>>>>> interpreted as an update in changelog view. >>>>>>>>> >>>>>>>>> To be perfectly honest, I'm confused about all of these details -- >>>>>>>>> from my read, the spec does not indicate this relationship between >>>>>>>>> identifier-fields and equality_ids in equality delete files ( >>>>>>>>> https://iceberg.apache.org/spec/#equality-delete-files), but I >>>>>>>>> think that is the way Flink works. Flink itself seems to have even >>>>>>>>> more >>>>>>>>> limitations -- no partition transforms are allowed, and all partition >>>>>>>>> columns must be a subset of the identifier fields. Is that just a >>>>>>>>> Flink >>>>>>>>> limitation, or is that the intended behavior in the spec? (Or maybe >>>>>>>>> user-error on my part?) Those seem like very reasonable limitations, >>>>>>>>> from >>>>>>>>> an implementation point-of-view. But OTOH, as a user, this seems to >>>>>>>>> be >>>>>>>>> directly contrary to some of the promises of Iceberg. >>>>>>>>> >>>>>>>>> Its easy to see if a table already has equality deletes in it, by >>>>>>>>> looking at the metadata. But is there any way to indicate that a >>>>>>>>> table (or >>>>>>>>> branch of a table) _must not_ have equality deletes added to it? >>>>>>>>> >>>>>>>>> If that were possible, it seems like we could support both use >>>>>>>>> cases. We could continue to optimize for the streaming ingestion use >>>>>>>>> cases >>>>>>>>> using equality deletes. But we could also build more optimizations >>>>>>>>> into >>>>>>>>> the "non-streaming-ingestion" branches. And we could document the >>>>>>>>> tradeoff >>>>>>>>> so it is much clearer to end users. >>>>>>>>> >>>>>>>>> To maintain compatibility, I suppose that the change would be that >>>>>>>>> equality deletes continue to be allowed by default, but we'd add a new >>>>>>>>> field to indicate that for some tables (or branches of a table), >>>>>>>>> equality >>>>>>>>> deletes would not be allowed. And it would be an error for an engine >>>>>>>>> to >>>>>>>>> make an update which added an equality delete to such a table. >>>>>>>>> >>>>>>>>> Maybe that change would even be possible in V3. >>>>>>>>> >>>>>>>>> And if all the performance improvements to equality deletes make >>>>>>>>> this a moot point, we could drop the field in v4. But it seems like a >>>>>>>>> mistake to both limit the non-streaming use-case AND have confusing >>>>>>>>> limitations for the end-user in the meantime. >>>>>>>>> >>>>>>>>> I would happily be corrected about my understanding of all of the >>>>>>>>> above. >>>>>>>>> >>>>>>>>> thanks! >>>>>>>>> Imran >>>>>>>>> >>>>>>>>> On Tue, Nov 5, 2024 at 9:16 AM Bryan Keller <brya...@gmail.com> >>>>>>>>> wrote: >>>>>>>>> >>>>>>>>>> I also feel we should keep equality deletes until we have an >>>>>>>>>> alternative solution for streaming updates/deletes. >>>>>>>>>> >>>>>>>>>> -Bryan >>>>>>>>>> >>>>>>>>>> On Nov 4, 2024, at 8:33 AM, Péter Váry < >>>>>>>>>> peter.vary.apa...@gmail.com> wrote: >>>>>>>>>> >>>>>>>>>> Well, it seems like I'm a little late, so most of the arguments >>>>>>>>>> are voiced. >>>>>>>>>> >>>>>>>>>> I agree that we should not deprecate the equality deletes until >>>>>>>>>> we have a replacement feature. >>>>>>>>>> I think one of the big advantages of Iceberg is that it supports >>>>>>>>>> batch processing and streaming ingestion too. >>>>>>>>>> For streaming ingestion we need a way to update existing data in >>>>>>>>>> a performant way, but restricting deletes for the primary keys seems >>>>>>>>>> like >>>>>>>>>> enough from the streaming perspective. >>>>>>>>>> >>>>>>>>>> Equality deletes allow a very wide range of applications, which >>>>>>>>>> we might be able to narrow down a bit, but still keep useful. So if >>>>>>>>>> we want >>>>>>>>>> to go down this road, we need to start collecting the requirements. >>>>>>>>>> >>>>>>>>>> Thanks, >>>>>>>>>> Peter >>>>>>>>>> >>>>>>>>>> Shani Elharrar <sh...@upsolver.com.invalid> ezt írta (időpont: >>>>>>>>>> 2024. nov. 1., P, 19:22): >>>>>>>>>> >>>>>>>>>>> I understand how it makes sense for batch jobs, but it damages >>>>>>>>>>> stream jobs, using equality deletes works much better for streaming >>>>>>>>>>> (which >>>>>>>>>>> have a strict SLA for delays), and in order to decrease the >>>>>>>>>>> performance >>>>>>>>>>> penalty - systems can rewrite the equality deletes to positional >>>>>>>>>>> deletes. >>>>>>>>>>> >>>>>>>>>>> Shani. >>>>>>>>>>> >>>>>>>>>>> On 1 Nov 2024, at 20:06, Steven Wu <stevenz...@gmail.com> wrote: >>>>>>>>>>> >>>>>>>>>>> >>>>>>>>>>> Fundamentally, it is very difficult to write position deletes >>>>>>>>>>> with concurrent writers and conflicts for batch jobs too, as the >>>>>>>>>>> inverted >>>>>>>>>>> index may become invalid/stale. >>>>>>>>>>> >>>>>>>>>>> The position deletes are created during the write phase. But >>>>>>>>>>> conflicts are only detected at the commit stage. I assume the batch >>>>>>>>>>> job >>>>>>>>>>> should fail in this case. >>>>>>>>>>> >>>>>>>>>>> On Fri, Nov 1, 2024 at 10:57 AM Steven Wu <stevenz...@gmail.com> >>>>>>>>>>> wrote: >>>>>>>>>>> >>>>>>>>>>>> Shani, >>>>>>>>>>>> >>>>>>>>>>>> That is a good point. It is certainly a limitation for the >>>>>>>>>>>> Flink job to track the inverted index internally (which is what I >>>>>>>>>>>> had in >>>>>>>>>>>> mind). It can't be shared/synchronized with other Flink jobs or >>>>>>>>>>>> other >>>>>>>>>>>> engines writing to the same table. >>>>>>>>>>>> >>>>>>>>>>>> Thanks, >>>>>>>>>>>> Steven >>>>>>>>>>>> >>>>>>>>>>>> On Fri, Nov 1, 2024 at 10:50 AM Shani Elharrar >>>>>>>>>>>> <sh...@upsolver.com.invalid> wrote: >>>>>>>>>>>> >>>>>>>>>>>>> Even if Flink can create this state, it would have to be >>>>>>>>>>>>> maintained against the Iceberg table, we wouldn't like duplicates >>>>>>>>>>>>> (keys) if >>>>>>>>>>>>> other systems / users update the table (e.g manual insert / >>>>>>>>>>>>> updates using >>>>>>>>>>>>> DML). >>>>>>>>>>>>> >>>>>>>>>>>>> Shani. >>>>>>>>>>>>> >>>>>>>>>>>>> On 1 Nov 2024, at 18:32, Steven Wu <stevenz...@gmail.com> >>>>>>>>>>>>> wrote: >>>>>>>>>>>>> >>>>>>>>>>>>> >>>>>>>>>>>>> > Add support for inverted indexes to reduce the cost of >>>>>>>>>>>>> position lookup. This is fairly tricky to implement for streaming >>>>>>>>>>>>> use cases >>>>>>>>>>>>> without an external system. >>>>>>>>>>>>> >>>>>>>>>>>>> Anton, that is also what I was saying earlier. In Flink, the >>>>>>>>>>>>> inverted index of (key, committed data files) can be tracked in >>>>>>>>>>>>> Flink state. >>>>>>>>>>>>> >>>>>>>>>>>>> On Fri, Nov 1, 2024 at 2:16 AM Anton Okolnychyi < >>>>>>>>>>>>> aokolnyc...@gmail.com> wrote: >>>>>>>>>>>>> >>>>>>>>>>>>>> I was a bit skeptical when we were adding equality deletes, >>>>>>>>>>>>>> but nothing beats their performance during writes. We have to >>>>>>>>>>>>>> find an >>>>>>>>>>>>>> alternative before deprecating. >>>>>>>>>>>>>> >>>>>>>>>>>>>> We are doing a lot of work to improve streaming, like >>>>>>>>>>>>>> reducing the cost of commits, enabling a large (potentially >>>>>>>>>>>>>> infinite) >>>>>>>>>>>>>> number of snapshots, changelog reads, and so on. It is a project >>>>>>>>>>>>>> goal to >>>>>>>>>>>>>> excel in streaming. >>>>>>>>>>>>>> >>>>>>>>>>>>>> I was going to focus on equality deletes after completing the >>>>>>>>>>>>>> DV work. I believe we have these options: >>>>>>>>>>>>>> >>>>>>>>>>>>>> - Revisit the existing design of equality deletes (e.g. add >>>>>>>>>>>>>> more restrictions, improve compaction, offer new writers). >>>>>>>>>>>>>> - Standardize on the view-based approach [1] to handle >>>>>>>>>>>>>> streaming upserts and CDC use cases, potentially making this >>>>>>>>>>>>>> part of the >>>>>>>>>>>>>> spec. >>>>>>>>>>>>>> - Add support for inverted indexes to reduce the cost of >>>>>>>>>>>>>> position lookup. This is fairly tricky to implement for >>>>>>>>>>>>>> streaming use cases >>>>>>>>>>>>>> without an external system. Our runtime filtering in Spark today >>>>>>>>>>>>>> is >>>>>>>>>>>>>> equivalent to looking up positions in an inverted index >>>>>>>>>>>>>> represented by >>>>>>>>>>>>>> another Iceberg table. That may still not be enough for some >>>>>>>>>>>>>> streaming use >>>>>>>>>>>>>> cases. >>>>>>>>>>>>>> >>>>>>>>>>>>>> [1] - https://www.tabular.io/blog/hello-world-of-cdc/ >>>>>>>>>>>>>> >>>>>>>>>>>>>> - Anton >>>>>>>>>>>>>> >>>>>>>>>>>>>> чт, 31 жовт. 2024 р. о 21:31 Micah Kornfield < >>>>>>>>>>>>>> emkornfi...@gmail.com> пише: >>>>>>>>>>>>>> >>>>>>>>>>>>>>> I agree that equality deletes have their place in >>>>>>>>>>>>>>> streaming. I think the ultimate decision here is how >>>>>>>>>>>>>>> opinionated >>>>>>>>>>>>>>> Iceberg wants to be on its use-cases. If it really wants to >>>>>>>>>>>>>>> stick to its >>>>>>>>>>>>>>> origins of "slow moving data", then removing equality deletes >>>>>>>>>>>>>>> would be >>>>>>>>>>>>>>> inline with this. I think the other high level question is how >>>>>>>>>>>>>>> much we >>>>>>>>>>>>>>> allow for partially compatible features (the row lineage >>>>>>>>>>>>>>> use-case feature >>>>>>>>>>>>>>> was explicitly approved excluding equality deletes, and people >>>>>>>>>>>>>>> seemed OK >>>>>>>>>>>>>>> with it at the time. If all features need to work together, >>>>>>>>>>>>>>> then maybe we >>>>>>>>>>>>>>> need to rethink the design here so it can be forward compatible >>>>>>>>>>>>>>> with >>>>>>>>>>>>>>> equality deletes). >>>>>>>>>>>>>>> >>>>>>>>>>>>>>> I think one issue with equality deletes as stated in the >>>>>>>>>>>>>>> spec is that they are overly broad. I'd be interested if >>>>>>>>>>>>>>> people have any >>>>>>>>>>>>>>> use cases that differ, but I think one way of narrowing (and >>>>>>>>>>>>>>> probably a >>>>>>>>>>>>>>> necessary building block for building something better) the >>>>>>>>>>>>>>> specification >>>>>>>>>>>>>>> scope on equality deletes is to focus on upsert/Streaming >>>>>>>>>>>>>>> deletes. Two >>>>>>>>>>>>>>> proposals in this regard are: >>>>>>>>>>>>>>> >>>>>>>>>>>>>>> 1. Require that equality deletes can only correspond to >>>>>>>>>>>>>>> unique identifiers for the table. >>>>>>>>>>>>>>> 2. Consider requiring that for equality deletes on >>>>>>>>>>>>>>> partitioned tables, that the primary key must contain a >>>>>>>>>>>>>>> partition column (I >>>>>>>>>>>>>>> believe Flink at least already does this). It is less clear to >>>>>>>>>>>>>>> me that >>>>>>>>>>>>>>> this would meet all existing use-cases. But having this would >>>>>>>>>>>>>>> allow for >>>>>>>>>>>>>>> better incremental data-structures, which could then be >>>>>>>>>>>>>>> partition based. >>>>>>>>>>>>>>> >>>>>>>>>>>>>>> Narrow scope to unique identifiers would allow for further >>>>>>>>>>>>>>> building blocks already mentioned, like a secondary index >>>>>>>>>>>>>>> (possible via LSM >>>>>>>>>>>>>>> tree), that would allow for better performance overall. >>>>>>>>>>>>>>> >>>>>>>>>>>>>>> I generally agree with the sentiment that we shouldn't >>>>>>>>>>>>>>> deprecate them until there is a viable replacement. With all >>>>>>>>>>>>>>> due respect >>>>>>>>>>>>>>> to my employer, let's not fall into the Google trap [1] :) >>>>>>>>>>>>>>> >>>>>>>>>>>>>>> Cheers, >>>>>>>>>>>>>>> Micah >>>>>>>>>>>>>>> >>>>>>>>>>>>>>> [1] https://goomics.net/50/ >>>>>>>>>>>>>>> >>>>>>>>>>>>>>> >>>>>>>>>>>>>>> >>>>>>>>>>>>>>> On Thu, Oct 31, 2024 at 12:35 PM Alexander Jo < >>>>>>>>>>>>>>> alex...@starburstdata.com> wrote: >>>>>>>>>>>>>>> >>>>>>>>>>>>>>>> Hey all, >>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>> Just to throw my 2 cents in, I agree with Steven and others >>>>>>>>>>>>>>>> that we do need some kind of replacement before deprecating >>>>>>>>>>>>>>>> equality >>>>>>>>>>>>>>>> deletes. >>>>>>>>>>>>>>>> They certainly have their problems, and do significantly >>>>>>>>>>>>>>>> increase complexity as they are now, but the writing of >>>>>>>>>>>>>>>> position deletes is >>>>>>>>>>>>>>>> too expensive for certain pipelines. >>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>> We've been investigating using equality deletes for some of >>>>>>>>>>>>>>>> our workloads at Starburst, the key advantage we were hoping >>>>>>>>>>>>>>>> to leverage is >>>>>>>>>>>>>>>> cheap, effectively random access lookup deletes. >>>>>>>>>>>>>>>> Say you have a UUID column that's unique in a table and >>>>>>>>>>>>>>>> want to delete a row by UUID. With position deletes each >>>>>>>>>>>>>>>> delete is >>>>>>>>>>>>>>>> expensive without an index on that UUID. >>>>>>>>>>>>>>>> With equality deletes each delete is cheap and while >>>>>>>>>>>>>>>> reads/compaction is expensive but when updates are frequent >>>>>>>>>>>>>>>> and reads are >>>>>>>>>>>>>>>> sporadic that's a reasonable tradeoff. >>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>> Pretty much what Jason and Steven have already said. >>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>> Maybe there are some incremental improvements on equality >>>>>>>>>>>>>>>> deletes or tips from similar systems that might alleviate some >>>>>>>>>>>>>>>> of their >>>>>>>>>>>>>>>> problems? >>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>> - Alex Jo >>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>> On Thu, Oct 31, 2024 at 10:58 AM Steven Wu < >>>>>>>>>>>>>>>> stevenz...@gmail.com> wrote: >>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>> We probably all agree with the downside of equality >>>>>>>>>>>>>>>>> deletes: it postpones all the work on the read path. >>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>> In theory, we can implement position deletes only in the >>>>>>>>>>>>>>>>> Flink streaming writer. It would require the tracking of last >>>>>>>>>>>>>>>>> committed >>>>>>>>>>>>>>>>> data files per key, which can be stored in Flink state >>>>>>>>>>>>>>>>> (checkpointed). This >>>>>>>>>>>>>>>>> is obviously quite expensive/challenging, but possible. >>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>> I like to echo one benefit of equality deletes that Russel >>>>>>>>>>>>>>>>> called out in the original email. Equality deletes would never >>>>>>>>>>>>>>>>> have conflicts. that is important for streaming writers >>>>>>>>>>>>>>>>> (Flink, Kafka >>>>>>>>>>>>>>>>> connect, ...) that commit frequently (minutes or less). >>>>>>>>>>>>>>>>> Assume Flink can >>>>>>>>>>>>>>>>> write position deletes only and commit every 2 minutes. The >>>>>>>>>>>>>>>>> long-running >>>>>>>>>>>>>>>>> nature of streaming jobs can cause frequent commit conflicts >>>>>>>>>>>>>>>>> with >>>>>>>>>>>>>>>>> background delete compaction jobs. >>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>> Overall, the streaming upsert write is not a well solved >>>>>>>>>>>>>>>>> problem in Iceberg. This probably affects all streaming >>>>>>>>>>>>>>>>> engines (Flink, >>>>>>>>>>>>>>>>> Kafka connect, Spark streaming, ...). We need to come up with >>>>>>>>>>>>>>>>> some better >>>>>>>>>>>>>>>>> alternatives before we can deprecate equality deletes. >>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>> On Thu, Oct 31, 2024 at 8:38 AM Russell Spitzer < >>>>>>>>>>>>>>>>> russell.spit...@gmail.com> wrote: >>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>> For users of Equality Deletes, what are the key >>>>>>>>>>>>>>>>>> benefits to Equality Deletes that you would like to preserve >>>>>>>>>>>>>>>>>> and could you >>>>>>>>>>>>>>>>>> please share some concrete examples of the queries you want >>>>>>>>>>>>>>>>>> to run (and the >>>>>>>>>>>>>>>>>> schemas and data sizes you would like to run them against) >>>>>>>>>>>>>>>>>> and the >>>>>>>>>>>>>>>>>> latencies that would be acceptable? >>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>> On Thu, Oct 31, 2024 at 10:05 AM Jason Fine >>>>>>>>>>>>>>>>>> <ja...@upsolver.com.invalid> wrote: >>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>> Hi, >>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>> Representing Upsolver here, we also make use of Equality >>>>>>>>>>>>>>>>>>> Deletes to deliver high frequency low latency updates to >>>>>>>>>>>>>>>>>>> our clients at >>>>>>>>>>>>>>>>>>> scale. We have customers using them at scale and >>>>>>>>>>>>>>>>>>> demonstrating the need and >>>>>>>>>>>>>>>>>>> viability. We automate the process of converting them into >>>>>>>>>>>>>>>>>>> positional >>>>>>>>>>>>>>>>>>> deletes (or fully applying them) for more efficient engine >>>>>>>>>>>>>>>>>>> queries in the >>>>>>>>>>>>>>>>>>> background giving our users both low latency and good query >>>>>>>>>>>>>>>>>>> performance. >>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>> Equality Deletes were added since there isn't a good way >>>>>>>>>>>>>>>>>>> to solve frequent updates otherwise. It would require some >>>>>>>>>>>>>>>>>>> sort of index >>>>>>>>>>>>>>>>>>> keeping track of every record in the table (by a >>>>>>>>>>>>>>>>>>> predetermined PK) and >>>>>>>>>>>>>>>>>>> maintaining such an index is a huge task that every tool >>>>>>>>>>>>>>>>>>> interested in this >>>>>>>>>>>>>>>>>>> would need to re-implement. It also becomes a bottleneck >>>>>>>>>>>>>>>>>>> limiting table >>>>>>>>>>>>>>>>>>> sizes. >>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>> I don't think they should be removed without providing >>>>>>>>>>>>>>>>>>> an alternative. Positional Deletes have a different >>>>>>>>>>>>>>>>>>> performance profile >>>>>>>>>>>>>>>>>>> inherently, requiring more upfront work proportional to the >>>>>>>>>>>>>>>>>>> table size. >>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>> On Thu, Oct 31, 2024 at 2:45 PM Jean-Baptiste Onofré < >>>>>>>>>>>>>>>>>>> j...@nanthrax.net> wrote: >>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>> Hi Russell >>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>> Thanks for the nice writeup and the proposal. >>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>> I agree with your analysis, and I have the same >>>>>>>>>>>>>>>>>>>> feeling. However, I >>>>>>>>>>>>>>>>>>>> think there are more than Flink that write equality >>>>>>>>>>>>>>>>>>>> delete files. So, >>>>>>>>>>>>>>>>>>>> I agree to deprecate in V3, but maybe be more >>>>>>>>>>>>>>>>>>>> "flexible" about removal >>>>>>>>>>>>>>>>>>>> in V4 in order to give time to engines to update. >>>>>>>>>>>>>>>>>>>> I think that by deprecating equality deletes, we are >>>>>>>>>>>>>>>>>>>> clearly focusing >>>>>>>>>>>>>>>>>>>> on read performance and "consistency" (more than >>>>>>>>>>>>>>>>>>>> write). It's not >>>>>>>>>>>>>>>>>>>> necessarily a bad thing but the streaming platform and >>>>>>>>>>>>>>>>>>>> data ingestion >>>>>>>>>>>>>>>>>>>> platforms will be probably concerned about that (by >>>>>>>>>>>>>>>>>>>> using positional >>>>>>>>>>>>>>>>>>>> deletes, they will have to scan/read all datafiles to >>>>>>>>>>>>>>>>>>>> find the >>>>>>>>>>>>>>>>>>>> position, so painful). >>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>> So, to summarize: >>>>>>>>>>>>>>>>>>>> 1. Agree to deprecate equality deletes, but -1 to >>>>>>>>>>>>>>>>>>>> commit any target >>>>>>>>>>>>>>>>>>>> for deletion before having a clear path for streaming >>>>>>>>>>>>>>>>>>>> platforms >>>>>>>>>>>>>>>>>>>> (Flink, Beam, ...) >>>>>>>>>>>>>>>>>>>> 2. In the meantime (during the deprecation period), I >>>>>>>>>>>>>>>>>>>> propose to >>>>>>>>>>>>>>>>>>>> explore possible improvements for streaming platforms >>>>>>>>>>>>>>>>>>>> (maybe finding a >>>>>>>>>>>>>>>>>>>> way to avoid full data files scan, ...) >>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>> Thanks ! >>>>>>>>>>>>>>>>>>>> Regards >>>>>>>>>>>>>>>>>>>> JB >>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>> On Wed, Oct 30, 2024 at 10:06 PM Russell Spitzer >>>>>>>>>>>>>>>>>>>> <russell.spit...@gmail.com> wrote: >>>>>>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>>>>>> > Background: >>>>>>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>>>>>> > 1) Position Deletes >>>>>>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>>>>>> > Writers determine what rows are deleted and mark them >>>>>>>>>>>>>>>>>>>> in a 1 for 1 representation. With delete vectors this >>>>>>>>>>>>>>>>>>>> means every data file >>>>>>>>>>>>>>>>>>>> has at most 1 delete vector that it is read in conjunction >>>>>>>>>>>>>>>>>>>> with to excise >>>>>>>>>>>>>>>>>>>> deleted rows. Reader overhead is more or less constant and >>>>>>>>>>>>>>>>>>>> is very >>>>>>>>>>>>>>>>>>>> predictable. >>>>>>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>>>>>> > The main cost of this mode is that deletes must be >>>>>>>>>>>>>>>>>>>> determined at write time which is expensive and can be >>>>>>>>>>>>>>>>>>>> more difficult for >>>>>>>>>>>>>>>>>>>> conflict resolution >>>>>>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>>>>>> > 2) Equality Deletes >>>>>>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>>>>>> > Writers write out reference to what values are >>>>>>>>>>>>>>>>>>>> deleted (in a partition or globally). There can be an >>>>>>>>>>>>>>>>>>>> unlimited number of >>>>>>>>>>>>>>>>>>>> equality deletes and they all must be checked for every >>>>>>>>>>>>>>>>>>>> data file that is >>>>>>>>>>>>>>>>>>>> read. The cost of determining deleted rows is essentially >>>>>>>>>>>>>>>>>>>> given to the >>>>>>>>>>>>>>>>>>>> reader. >>>>>>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>>>>>> > Conflicts almost never happen since data files are >>>>>>>>>>>>>>>>>>>> not actually changed and there is almost no cost to the >>>>>>>>>>>>>>>>>>>> writer to generate >>>>>>>>>>>>>>>>>>>> these. Almost all costs related to equality deletes are >>>>>>>>>>>>>>>>>>>> passed on to the >>>>>>>>>>>>>>>>>>>> reader. >>>>>>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>>>>>> > Proposal: >>>>>>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>>>>>> > Equality deletes are, in my opinion, unsustainable >>>>>>>>>>>>>>>>>>>> and we should work on deprecating and removing them from >>>>>>>>>>>>>>>>>>>> the specification. >>>>>>>>>>>>>>>>>>>> At this time, I know of only one engine (Apache Flink) >>>>>>>>>>>>>>>>>>>> which produces these >>>>>>>>>>>>>>>>>>>> deletes but almost all engines have implementations to >>>>>>>>>>>>>>>>>>>> read them. The cost >>>>>>>>>>>>>>>>>>>> of implementing equality deletes on the read path is >>>>>>>>>>>>>>>>>>>> difficult and >>>>>>>>>>>>>>>>>>>> unpredictable in terms of memory usage and compute >>>>>>>>>>>>>>>>>>>> complexity. We’ve had >>>>>>>>>>>>>>>>>>>> suggestions of implementing rocksdb inorder to handle ever >>>>>>>>>>>>>>>>>>>> growing sets of >>>>>>>>>>>>>>>>>>>> equality deletes which in my opinion shows that we are >>>>>>>>>>>>>>>>>>>> going down the wrong >>>>>>>>>>>>>>>>>>>> path. >>>>>>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>>>>>> > Outside of performance, Equality deletes are also >>>>>>>>>>>>>>>>>>>> difficult to use in conjunction with many other features. >>>>>>>>>>>>>>>>>>>> For example, any >>>>>>>>>>>>>>>>>>>> features requiring CDC or Row lineage are basically >>>>>>>>>>>>>>>>>>>> impossible when >>>>>>>>>>>>>>>>>>>> equality deletes are in use. When Equality deletes are >>>>>>>>>>>>>>>>>>>> present, the state >>>>>>>>>>>>>>>>>>>> of the table can only be determined with a full scan >>>>>>>>>>>>>>>>>>>> making it difficult to >>>>>>>>>>>>>>>>>>>> update differential structures. This means materialized >>>>>>>>>>>>>>>>>>>> views or indexes >>>>>>>>>>>>>>>>>>>> need to essentially be fully rebuilt whenever an equality >>>>>>>>>>>>>>>>>>>> delete is added >>>>>>>>>>>>>>>>>>>> to the table. >>>>>>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>>>>>> > Equality deletes essentially remove complexity from >>>>>>>>>>>>>>>>>>>> the write side but then add what I believe is an >>>>>>>>>>>>>>>>>>>> unacceptable level of >>>>>>>>>>>>>>>>>>>> complexity to the read side. >>>>>>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>>>>>> > Because of this I suggest we deprecate Equality >>>>>>>>>>>>>>>>>>>> Deletes in V3 and slate them for full removal from the >>>>>>>>>>>>>>>>>>>> Iceberg Spec in V4. >>>>>>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>>>>>> > I know this is a big change and compatibility >>>>>>>>>>>>>>>>>>>> breakage so I would like to introduce this idea to the >>>>>>>>>>>>>>>>>>>> community and >>>>>>>>>>>>>>>>>>>> solicit feedback from all stakeholders. I am very flexible >>>>>>>>>>>>>>>>>>>> on this issue >>>>>>>>>>>>>>>>>>>> and would like to hear the best issues both for and >>>>>>>>>>>>>>>>>>>> against removal of >>>>>>>>>>>>>>>>>>>> Equality Deletes. >>>>>>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>>>>>> > Thanks everyone for your time, >>>>>>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>>>>>> > Russ Spitzer >>>>>>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>> -- >>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>> *Jason Fine* >>>>>>>>>>>>>>>>>>> Chief Software Architect >>>>>>>>>>>>>>>>>>> ja...@upsolver.com | www.upsolver.com >>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>> >>>>>>>>>>
