I don’t think it’s a problem while an alternative is explored (the JDK itself does that pretty often). So it’s up to the community: of course I’m against removing it without solid alternative, but deprecation is fine imho.
Regards JB Le mar. 19 nov. 2024 à 12:19, Ajantha Bhat <ajanthab...@gmail.com> a écrit : > - ok for deprecate equality deletes >> - not ok to remove it > > > @JB: I don't think it is a good idea to use deprecated functionality in > the new feature development. > Hence, my specific question was about kafka connect upsert operation. > > @Manu: I meant the delta writers for kafka connect Iceberg sink (which in > turn used for upsetting the CDC records) > https://github.com/apache/iceberg/issues/10842 > > > - Ajantha > > > > On Tue, Nov 19, 2024 at 3:08 PM Manu Zhang <owenzhang1...@gmail.com> > wrote: > >> I second Anton's proposal to standardize on a view-based approach to >> handle CDC cases. >> Actually, it's already been explored in detail[1] by Jack before. >> >> [1] Improving Change Data Capture Use Case for Apache Iceberg >> <https://docs.google.com/document/d/1kyyJp4masbd1FrIKUHF1ED_z1hTARL8bNoKCgb7fhSQ/edit?tab=t.0#heading=h.94xnx4qg3bnt> >> >> >> On Tue, Nov 19, 2024 at 4:16 PM Jean-Baptiste Onofré <j...@nanthrax.net> >> wrote: >> >>> My proposal is the following (already expressed): >>> - ok for deprecate equality deletes >>> - not ok to remove it >>> - work on position deletes improvements to address streaming use cases. >>> I think we should explore different approaches. Personally I think a >>> possible approach would be to find index way to data files to avoid full >>> scan to find row position. >>> >>> My $0.01 :) >>> >>> Regards >>> JB >>> >>> Le mar. 19 nov. 2024 à 07:53, Ajantha Bhat <ajanthab...@gmail.com> a >>> écrit : >>> >>>> Hi, What's the conclusion on this thread? >>>> >>>> Users are looking for Upsert (CDC) support for OSS Iceberg >>>> kafka connect sink. >>>> We only support appends at the moment. Can we go ahead and implement >>>> the upserts using equality deletes? >>>> >>>> >>>> - Ajantha >>>> >>>> On Sun, Nov 10, 2024 at 11:56 AM Vignesh <vignesh.v...@gmail.com> >>>> wrote: >>>> >>>>> Hi, >>>>> I am reading about iceberg and am quite new to this. >>>>> This puffin would be an index from key to data file. Other use cases >>>>> of Puffin, such as statistics are at a per file level if I understand >>>>> correctly. >>>>> >>>>> Where would the puffin about key->data file be stored? It is a >>>>> property of the entire table. >>>>> >>>>> Thanks, >>>>> Vignesh. >>>>> >>>>> >>>>> On Sat, Nov 9, 2024 at 2:17 AM Shani Elharrar >>>>> <sh...@upsolver.com.invalid> wrote: >>>>> >>>>>> JB, this is what we do, we write Equality Deletes and periodically >>>>>> convert them to Positional Deletes. >>>>>> >>>>>> We could probably index the keys, maybe partially index using bloom >>>>>> filters, the best would be to put those bloom filters inside puffin. >>>>>> >>>>>> Shani. >>>>>> >>>>>> On 9 Nov 2024, at 11:11, Jean-Baptiste Onofré <j...@nanthrax.net> >>>>>> wrote: >>>>>> >>>>>> >>>>>> Hi, >>>>>> >>>>>> I agree with Peter here, and I would say that it would be an issue >>>>>> for multi-engine support. >>>>>> >>>>>> I think, as I already mentioned with others, we should explore an >>>>>> alternative. >>>>>> As the main issue is the datafile scan in streaming context, maybe we >>>>>> could find a way to "index"/correlate for positional deletes with limited >>>>>> scanning. >>>>>> I will think again about that :) >>>>>> >>>>>> Regards >>>>>> JB >>>>>> >>>>>> On Sat, Nov 9, 2024 at 6:48 AM Péter Váry < >>>>>> peter.vary.apa...@gmail.com> wrote: >>>>>> >>>>>>> Hi Imran, >>>>>>> >>>>>>> I don't think it's a good idea to start creating multiple types of >>>>>>> Iceberg tables. Iceberg's main selling point is compatibility between >>>>>>> engines. If we don't have readers and writers for all types of tables, >>>>>>> then >>>>>>> we remove compatibility from the equation and engine specific formats >>>>>>> always win. OTOH, if we write readers and writers for all types of >>>>>>> tables >>>>>>> then we are back on square one. >>>>>>> >>>>>>> Identifier fields are a table schema concept and used in many cases >>>>>>> during query planning and execution. This is why they are defined as >>>>>>> part >>>>>>> of the SQL spec, and this is why Iceberg defines them as well. One use >>>>>>> case >>>>>>> is where they can be used to merge deletes (independently of how they >>>>>>> are >>>>>>> manifested) and subsequent inserts, into updates. >>>>>>> >>>>>>> Flink SQL doesn't allow creating tables with partition transforms, >>>>>>> so no new table could be created by Flink SQL using transforms, but >>>>>>> tables >>>>>>> created by other engines could still be used (both read an write). Also >>>>>>> you >>>>>>> can create such tables in Flink using the Java API. >>>>>>> >>>>>>> Requiring partition columns be part of the identifier fields is >>>>>>> coming from the practical consideration, that you want to limit the >>>>>>> scope >>>>>>> of the equality deletes as much as possible. Otherwise all of the >>>>>>> equality >>>>>>> deletes should be table global, and they should be read by every >>>>>>> reader. We >>>>>>> could write those, we just decided that we don't want to allow the user >>>>>>> to >>>>>>> do this, as it is most cases a bad idea. >>>>>>> >>>>>>> I hope this helps, >>>>>>> Peter >>>>>>> >>>>>>> On Fri, Nov 8, 2024, 22:01 Imran Rashid <iras...@cloudera.com.invalid> >>>>>>> wrote: >>>>>>> >>>>>>>> I'm not down in the weeds at all myself on implementation details, >>>>>>>> so forgive me if I'm wrong about the details here. >>>>>>>> >>>>>>>> I can see all the viewpoints -- both that equality deletes enable >>>>>>>> some use cases, but also make others far more difficult. What >>>>>>>> surprised me >>>>>>>> the most is that Iceberg does not provide a way to distinguish these >>>>>>>> two >>>>>>>> table "types". >>>>>>>> >>>>>>>> At first, I thought the presence of an identifier-field ( >>>>>>>> https://iceberg.apache.org/spec/#identifier-field-ids) indicated >>>>>>>> that the table was a target for equality deletes. But, then it turns >>>>>>>> out >>>>>>>> identifier-fields are also useful for changelog views even without >>>>>>>> equality >>>>>>>> deletes -- IIUC, they show that a delete + insert should actually be >>>>>>>> interpreted as an update in changelog view. >>>>>>>> >>>>>>>> To be perfectly honest, I'm confused about all of these details -- >>>>>>>> from my read, the spec does not indicate this relationship between >>>>>>>> identifier-fields and equality_ids in equality delete files ( >>>>>>>> https://iceberg.apache.org/spec/#equality-delete-files), but I >>>>>>>> think that is the way Flink works. Flink itself seems to have even >>>>>>>> more >>>>>>>> limitations -- no partition transforms are allowed, and all partition >>>>>>>> columns must be a subset of the identifier fields. Is that just a >>>>>>>> Flink >>>>>>>> limitation, or is that the intended behavior in the spec? (Or maybe >>>>>>>> user-error on my part?) Those seem like very reasonable limitations, >>>>>>>> from >>>>>>>> an implementation point-of-view. But OTOH, as a user, this seems to be >>>>>>>> directly contrary to some of the promises of Iceberg. >>>>>>>> >>>>>>>> Its easy to see if a table already has equality deletes in it, by >>>>>>>> looking at the metadata. But is there any way to indicate that a >>>>>>>> table (or >>>>>>>> branch of a table) _must not_ have equality deletes added to it? >>>>>>>> >>>>>>>> If that were possible, it seems like we could support both use >>>>>>>> cases. We could continue to optimize for the streaming ingestion use >>>>>>>> cases >>>>>>>> using equality deletes. But we could also build more optimizations >>>>>>>> into >>>>>>>> the "non-streaming-ingestion" branches. And we could document the >>>>>>>> tradeoff >>>>>>>> so it is much clearer to end users. >>>>>>>> >>>>>>>> To maintain compatibility, I suppose that the change would be that >>>>>>>> equality deletes continue to be allowed by default, but we'd add a new >>>>>>>> field to indicate that for some tables (or branches of a table), >>>>>>>> equality >>>>>>>> deletes would not be allowed. And it would be an error for an engine >>>>>>>> to >>>>>>>> make an update which added an equality delete to such a table. >>>>>>>> >>>>>>>> Maybe that change would even be possible in V3. >>>>>>>> >>>>>>>> And if all the performance improvements to equality deletes make >>>>>>>> this a moot point, we could drop the field in v4. But it seems like a >>>>>>>> mistake to both limit the non-streaming use-case AND have confusing >>>>>>>> limitations for the end-user in the meantime. >>>>>>>> >>>>>>>> I would happily be corrected about my understanding of all of the >>>>>>>> above. >>>>>>>> >>>>>>>> thanks! >>>>>>>> Imran >>>>>>>> >>>>>>>> On Tue, Nov 5, 2024 at 9:16 AM Bryan Keller <brya...@gmail.com> >>>>>>>> wrote: >>>>>>>> >>>>>>>>> I also feel we should keep equality deletes until we have an >>>>>>>>> alternative solution for streaming updates/deletes. >>>>>>>>> >>>>>>>>> -Bryan >>>>>>>>> >>>>>>>>> On Nov 4, 2024, at 8:33 AM, Péter Váry < >>>>>>>>> peter.vary.apa...@gmail.com> wrote: >>>>>>>>> >>>>>>>>> Well, it seems like I'm a little late, so most of the arguments >>>>>>>>> are voiced. >>>>>>>>> >>>>>>>>> I agree that we should not deprecate the equality deletes until we >>>>>>>>> have a replacement feature. >>>>>>>>> I think one of the big advantages of Iceberg is that it supports >>>>>>>>> batch processing and streaming ingestion too. >>>>>>>>> For streaming ingestion we need a way to update existing data in a >>>>>>>>> performant way, but restricting deletes for the primary keys seems >>>>>>>>> like >>>>>>>>> enough from the streaming perspective. >>>>>>>>> >>>>>>>>> Equality deletes allow a very wide range of applications, which we >>>>>>>>> might be able to narrow down a bit, but still keep useful. So if we >>>>>>>>> want to >>>>>>>>> go down this road, we need to start collecting the requirements. >>>>>>>>> >>>>>>>>> Thanks, >>>>>>>>> Peter >>>>>>>>> >>>>>>>>> Shani Elharrar <sh...@upsolver.com.invalid> ezt írta (időpont: >>>>>>>>> 2024. nov. 1., P, 19:22): >>>>>>>>> >>>>>>>>>> I understand how it makes sense for batch jobs, but it damages >>>>>>>>>> stream jobs, using equality deletes works much better for streaming >>>>>>>>>> (which >>>>>>>>>> have a strict SLA for delays), and in order to decrease the >>>>>>>>>> performance >>>>>>>>>> penalty - systems can rewrite the equality deletes to positional >>>>>>>>>> deletes. >>>>>>>>>> >>>>>>>>>> Shani. >>>>>>>>>> >>>>>>>>>> On 1 Nov 2024, at 20:06, Steven Wu <stevenz...@gmail.com> wrote: >>>>>>>>>> >>>>>>>>>> >>>>>>>>>> Fundamentally, it is very difficult to write position deletes >>>>>>>>>> with concurrent writers and conflicts for batch jobs too, as the >>>>>>>>>> inverted >>>>>>>>>> index may become invalid/stale. >>>>>>>>>> >>>>>>>>>> The position deletes are created during the write phase. But >>>>>>>>>> conflicts are only detected at the commit stage. I assume the batch >>>>>>>>>> job >>>>>>>>>> should fail in this case. >>>>>>>>>> >>>>>>>>>> On Fri, Nov 1, 2024 at 10:57 AM Steven Wu <stevenz...@gmail.com> >>>>>>>>>> wrote: >>>>>>>>>> >>>>>>>>>>> Shani, >>>>>>>>>>> >>>>>>>>>>> That is a good point. It is certainly a limitation for the Flink >>>>>>>>>>> job to track the inverted index internally (which is what I had in >>>>>>>>>>> mind). >>>>>>>>>>> It can't be shared/synchronized with other Flink jobs or other >>>>>>>>>>> engines >>>>>>>>>>> writing to the same table. >>>>>>>>>>> >>>>>>>>>>> Thanks, >>>>>>>>>>> Steven >>>>>>>>>>> >>>>>>>>>>> On Fri, Nov 1, 2024 at 10:50 AM Shani Elharrar >>>>>>>>>>> <sh...@upsolver.com.invalid> wrote: >>>>>>>>>>> >>>>>>>>>>>> Even if Flink can create this state, it would have to be >>>>>>>>>>>> maintained against the Iceberg table, we wouldn't like duplicates >>>>>>>>>>>> (keys) if >>>>>>>>>>>> other systems / users update the table (e.g manual insert / >>>>>>>>>>>> updates using >>>>>>>>>>>> DML). >>>>>>>>>>>> >>>>>>>>>>>> Shani. >>>>>>>>>>>> >>>>>>>>>>>> On 1 Nov 2024, at 18:32, Steven Wu <stevenz...@gmail.com> >>>>>>>>>>>> wrote: >>>>>>>>>>>> >>>>>>>>>>>> >>>>>>>>>>>> > Add support for inverted indexes to reduce the cost of >>>>>>>>>>>> position lookup. This is fairly tricky to implement for streaming >>>>>>>>>>>> use cases >>>>>>>>>>>> without an external system. >>>>>>>>>>>> >>>>>>>>>>>> Anton, that is also what I was saying earlier. In Flink, the >>>>>>>>>>>> inverted index of (key, committed data files) can be tracked in >>>>>>>>>>>> Flink state. >>>>>>>>>>>> >>>>>>>>>>>> On Fri, Nov 1, 2024 at 2:16 AM Anton Okolnychyi < >>>>>>>>>>>> aokolnyc...@gmail.com> wrote: >>>>>>>>>>>> >>>>>>>>>>>>> I was a bit skeptical when we were adding equality deletes, >>>>>>>>>>>>> but nothing beats their performance during writes. We have to >>>>>>>>>>>>> find an >>>>>>>>>>>>> alternative before deprecating. >>>>>>>>>>>>> >>>>>>>>>>>>> We are doing a lot of work to improve streaming, like reducing >>>>>>>>>>>>> the cost of commits, enabling a large (potentially infinite) >>>>>>>>>>>>> number of >>>>>>>>>>>>> snapshots, changelog reads, and so on. It is a project goal to >>>>>>>>>>>>> excel in >>>>>>>>>>>>> streaming. >>>>>>>>>>>>> >>>>>>>>>>>>> I was going to focus on equality deletes after completing the >>>>>>>>>>>>> DV work. I believe we have these options: >>>>>>>>>>>>> >>>>>>>>>>>>> - Revisit the existing design of equality deletes (e.g. add >>>>>>>>>>>>> more restrictions, improve compaction, offer new writers). >>>>>>>>>>>>> - Standardize on the view-based approach [1] to handle >>>>>>>>>>>>> streaming upserts and CDC use cases, potentially making this part >>>>>>>>>>>>> of the >>>>>>>>>>>>> spec. >>>>>>>>>>>>> - Add support for inverted indexes to reduce the cost of >>>>>>>>>>>>> position lookup. This is fairly tricky to implement for streaming >>>>>>>>>>>>> use cases >>>>>>>>>>>>> without an external system. Our runtime filtering in Spark today >>>>>>>>>>>>> is >>>>>>>>>>>>> equivalent to looking up positions in an inverted index >>>>>>>>>>>>> represented by >>>>>>>>>>>>> another Iceberg table. That may still not be enough for some >>>>>>>>>>>>> streaming use >>>>>>>>>>>>> cases. >>>>>>>>>>>>> >>>>>>>>>>>>> [1] - https://www.tabular.io/blog/hello-world-of-cdc/ >>>>>>>>>>>>> >>>>>>>>>>>>> - Anton >>>>>>>>>>>>> >>>>>>>>>>>>> чт, 31 жовт. 2024 р. о 21:31 Micah Kornfield < >>>>>>>>>>>>> emkornfi...@gmail.com> пише: >>>>>>>>>>>>> >>>>>>>>>>>>>> I agree that equality deletes have their place in streaming. >>>>>>>>>>>>>> I think the ultimate decision here is how opinionated Iceberg >>>>>>>>>>>>>> wants to be >>>>>>>>>>>>>> on its use-cases. If it really wants to stick to its origins of >>>>>>>>>>>>>> "slow >>>>>>>>>>>>>> moving data", then removing equality deletes would be inline >>>>>>>>>>>>>> with this. I >>>>>>>>>>>>>> think the other high level question is how much we allow for >>>>>>>>>>>>>> partially >>>>>>>>>>>>>> compatible features (the row lineage use-case feature was >>>>>>>>>>>>>> explicitly >>>>>>>>>>>>>> approved excluding equality deletes, and people seemed OK with >>>>>>>>>>>>>> it at the >>>>>>>>>>>>>> time. If all features need to work together, then maybe we need >>>>>>>>>>>>>> to rethink >>>>>>>>>>>>>> the design here so it can be forward compatible with equality >>>>>>>>>>>>>> deletes). >>>>>>>>>>>>>> >>>>>>>>>>>>>> I think one issue with equality deletes as stated in the spec >>>>>>>>>>>>>> is that they are overly broad. I'd be interested if people have >>>>>>>>>>>>>> any use >>>>>>>>>>>>>> cases that differ, but I think one way of narrowing (and >>>>>>>>>>>>>> probably a >>>>>>>>>>>>>> necessary building block for building something better) the >>>>>>>>>>>>>> specification >>>>>>>>>>>>>> scope on equality deletes is to focus on upsert/Streaming >>>>>>>>>>>>>> deletes. Two >>>>>>>>>>>>>> proposals in this regard are: >>>>>>>>>>>>>> >>>>>>>>>>>>>> 1. Require that equality deletes can only correspond to >>>>>>>>>>>>>> unique identifiers for the table. >>>>>>>>>>>>>> 2. Consider requiring that for equality deletes on >>>>>>>>>>>>>> partitioned tables, that the primary key must contain a >>>>>>>>>>>>>> partition column (I >>>>>>>>>>>>>> believe Flink at least already does this). It is less clear to >>>>>>>>>>>>>> me that >>>>>>>>>>>>>> this would meet all existing use-cases. But having this would >>>>>>>>>>>>>> allow for >>>>>>>>>>>>>> better incremental data-structures, which could then be >>>>>>>>>>>>>> partition based. >>>>>>>>>>>>>> >>>>>>>>>>>>>> Narrow scope to unique identifiers would allow for further >>>>>>>>>>>>>> building blocks already mentioned, like a secondary index >>>>>>>>>>>>>> (possible via LSM >>>>>>>>>>>>>> tree), that would allow for better performance overall. >>>>>>>>>>>>>> >>>>>>>>>>>>>> I generally agree with the sentiment that we shouldn't >>>>>>>>>>>>>> deprecate them until there is a viable replacement. With all >>>>>>>>>>>>>> due respect >>>>>>>>>>>>>> to my employer, let's not fall into the Google trap [1] :) >>>>>>>>>>>>>> >>>>>>>>>>>>>> Cheers, >>>>>>>>>>>>>> Micah >>>>>>>>>>>>>> >>>>>>>>>>>>>> [1] https://goomics.net/50/ >>>>>>>>>>>>>> >>>>>>>>>>>>>> >>>>>>>>>>>>>> >>>>>>>>>>>>>> On Thu, Oct 31, 2024 at 12:35 PM Alexander Jo < >>>>>>>>>>>>>> alex...@starburstdata.com> wrote: >>>>>>>>>>>>>> >>>>>>>>>>>>>>> Hey all, >>>>>>>>>>>>>>> >>>>>>>>>>>>>>> Just to throw my 2 cents in, I agree with Steven and others >>>>>>>>>>>>>>> that we do need some kind of replacement before deprecating >>>>>>>>>>>>>>> equality >>>>>>>>>>>>>>> deletes. >>>>>>>>>>>>>>> They certainly have their problems, and do significantly >>>>>>>>>>>>>>> increase complexity as they are now, but the writing of >>>>>>>>>>>>>>> position deletes is >>>>>>>>>>>>>>> too expensive for certain pipelines. >>>>>>>>>>>>>>> >>>>>>>>>>>>>>> We've been investigating using equality deletes for some of >>>>>>>>>>>>>>> our workloads at Starburst, the key advantage we were hoping to >>>>>>>>>>>>>>> leverage is >>>>>>>>>>>>>>> cheap, effectively random access lookup deletes. >>>>>>>>>>>>>>> Say you have a UUID column that's unique in a table and want >>>>>>>>>>>>>>> to delete a row by UUID. With position deletes each delete is >>>>>>>>>>>>>>> expensive >>>>>>>>>>>>>>> without an index on that UUID. >>>>>>>>>>>>>>> With equality deletes each delete is cheap and while >>>>>>>>>>>>>>> reads/compaction is expensive but when updates are frequent and >>>>>>>>>>>>>>> reads are >>>>>>>>>>>>>>> sporadic that's a reasonable tradeoff. >>>>>>>>>>>>>>> >>>>>>>>>>>>>>> Pretty much what Jason and Steven have already said. >>>>>>>>>>>>>>> >>>>>>>>>>>>>>> Maybe there are some incremental improvements on equality >>>>>>>>>>>>>>> deletes or tips from similar systems that might alleviate some >>>>>>>>>>>>>>> of their >>>>>>>>>>>>>>> problems? >>>>>>>>>>>>>>> >>>>>>>>>>>>>>> - Alex Jo >>>>>>>>>>>>>>> >>>>>>>>>>>>>>> On Thu, Oct 31, 2024 at 10:58 AM Steven Wu < >>>>>>>>>>>>>>> stevenz...@gmail.com> wrote: >>>>>>>>>>>>>>> >>>>>>>>>>>>>>>> We probably all agree with the downside of equality >>>>>>>>>>>>>>>> deletes: it postpones all the work on the read path. >>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>> In theory, we can implement position deletes only in the >>>>>>>>>>>>>>>> Flink streaming writer. It would require the tracking of last >>>>>>>>>>>>>>>> committed >>>>>>>>>>>>>>>> data files per key, which can be stored in Flink state >>>>>>>>>>>>>>>> (checkpointed). This >>>>>>>>>>>>>>>> is obviously quite expensive/challenging, but possible. >>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>> I like to echo one benefit of equality deletes that Russel >>>>>>>>>>>>>>>> called out in the original email. Equality deletes would never >>>>>>>>>>>>>>>> have conflicts. that is important for streaming writers >>>>>>>>>>>>>>>> (Flink, Kafka >>>>>>>>>>>>>>>> connect, ...) that commit frequently (minutes or less). Assume >>>>>>>>>>>>>>>> Flink can >>>>>>>>>>>>>>>> write position deletes only and commit every 2 minutes. The >>>>>>>>>>>>>>>> long-running >>>>>>>>>>>>>>>> nature of streaming jobs can cause frequent commit conflicts >>>>>>>>>>>>>>>> with >>>>>>>>>>>>>>>> background delete compaction jobs. >>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>> Overall, the streaming upsert write is not a well solved >>>>>>>>>>>>>>>> problem in Iceberg. This probably affects all streaming >>>>>>>>>>>>>>>> engines (Flink, >>>>>>>>>>>>>>>> Kafka connect, Spark streaming, ...). We need to come up with >>>>>>>>>>>>>>>> some better >>>>>>>>>>>>>>>> alternatives before we can deprecate equality deletes. >>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>> On Thu, Oct 31, 2024 at 8:38 AM Russell Spitzer < >>>>>>>>>>>>>>>> russell.spit...@gmail.com> wrote: >>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>> For users of Equality Deletes, what are the key >>>>>>>>>>>>>>>>> benefits to Equality Deletes that you would like to preserve >>>>>>>>>>>>>>>>> and could you >>>>>>>>>>>>>>>>> please share some concrete examples of the queries you want >>>>>>>>>>>>>>>>> to run (and the >>>>>>>>>>>>>>>>> schemas and data sizes you would like to run them against) >>>>>>>>>>>>>>>>> and the >>>>>>>>>>>>>>>>> latencies that would be acceptable? >>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>> On Thu, Oct 31, 2024 at 10:05 AM Jason Fine >>>>>>>>>>>>>>>>> <ja...@upsolver.com.invalid> wrote: >>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>> Hi, >>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>> Representing Upsolver here, we also make use of Equality >>>>>>>>>>>>>>>>>> Deletes to deliver high frequency low latency updates to our >>>>>>>>>>>>>>>>>> clients at >>>>>>>>>>>>>>>>>> scale. We have customers using them at scale and >>>>>>>>>>>>>>>>>> demonstrating the need and >>>>>>>>>>>>>>>>>> viability. We automate the process of converting them into >>>>>>>>>>>>>>>>>> positional >>>>>>>>>>>>>>>>>> deletes (or fully applying them) for more efficient engine >>>>>>>>>>>>>>>>>> queries in the >>>>>>>>>>>>>>>>>> background giving our users both low latency and good query >>>>>>>>>>>>>>>>>> performance. >>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>> Equality Deletes were added since there isn't a good way >>>>>>>>>>>>>>>>>> to solve frequent updates otherwise. It would require some >>>>>>>>>>>>>>>>>> sort of index >>>>>>>>>>>>>>>>>> keeping track of every record in the table (by a >>>>>>>>>>>>>>>>>> predetermined PK) and >>>>>>>>>>>>>>>>>> maintaining such an index is a huge task that every tool >>>>>>>>>>>>>>>>>> interested in this >>>>>>>>>>>>>>>>>> would need to re-implement. It also becomes a bottleneck >>>>>>>>>>>>>>>>>> limiting table >>>>>>>>>>>>>>>>>> sizes. >>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>> I don't think they should be removed without providing an >>>>>>>>>>>>>>>>>> alternative. Positional Deletes have a different performance >>>>>>>>>>>>>>>>>> profile >>>>>>>>>>>>>>>>>> inherently, requiring more upfront work proportional to the >>>>>>>>>>>>>>>>>> table size. >>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>> On Thu, Oct 31, 2024 at 2:45 PM Jean-Baptiste Onofré < >>>>>>>>>>>>>>>>>> j...@nanthrax.net> wrote: >>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>> Hi Russell >>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>> Thanks for the nice writeup and the proposal. >>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>> I agree with your analysis, and I have the same feeling. >>>>>>>>>>>>>>>>>>> However, I >>>>>>>>>>>>>>>>>>> think there are more than Flink that write equality >>>>>>>>>>>>>>>>>>> delete files. So, >>>>>>>>>>>>>>>>>>> I agree to deprecate in V3, but maybe be more "flexible" >>>>>>>>>>>>>>>>>>> about removal >>>>>>>>>>>>>>>>>>> in V4 in order to give time to engines to update. >>>>>>>>>>>>>>>>>>> I think that by deprecating equality deletes, we are >>>>>>>>>>>>>>>>>>> clearly focusing >>>>>>>>>>>>>>>>>>> on read performance and "consistency" (more than write). >>>>>>>>>>>>>>>>>>> It's not >>>>>>>>>>>>>>>>>>> necessarily a bad thing but the streaming platform and >>>>>>>>>>>>>>>>>>> data ingestion >>>>>>>>>>>>>>>>>>> platforms will be probably concerned about that (by >>>>>>>>>>>>>>>>>>> using positional >>>>>>>>>>>>>>>>>>> deletes, they will have to scan/read all datafiles to >>>>>>>>>>>>>>>>>>> find the >>>>>>>>>>>>>>>>>>> position, so painful). >>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>> So, to summarize: >>>>>>>>>>>>>>>>>>> 1. Agree to deprecate equality deletes, but -1 to commit >>>>>>>>>>>>>>>>>>> any target >>>>>>>>>>>>>>>>>>> for deletion before having a clear path for streaming >>>>>>>>>>>>>>>>>>> platforms >>>>>>>>>>>>>>>>>>> (Flink, Beam, ...) >>>>>>>>>>>>>>>>>>> 2. In the meantime (during the deprecation period), I >>>>>>>>>>>>>>>>>>> propose to >>>>>>>>>>>>>>>>>>> explore possible improvements for streaming platforms >>>>>>>>>>>>>>>>>>> (maybe finding a >>>>>>>>>>>>>>>>>>> way to avoid full data files scan, ...) >>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>> Thanks ! >>>>>>>>>>>>>>>>>>> Regards >>>>>>>>>>>>>>>>>>> JB >>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>> On Wed, Oct 30, 2024 at 10:06 PM Russell Spitzer >>>>>>>>>>>>>>>>>>> <russell.spit...@gmail.com> wrote: >>>>>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>>>>> > Background: >>>>>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>>>>> > 1) Position Deletes >>>>>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>>>>> > Writers determine what rows are deleted and mark them >>>>>>>>>>>>>>>>>>> in a 1 for 1 representation. With delete vectors this means >>>>>>>>>>>>>>>>>>> every data file >>>>>>>>>>>>>>>>>>> has at most 1 delete vector that it is read in conjunction >>>>>>>>>>>>>>>>>>> with to excise >>>>>>>>>>>>>>>>>>> deleted rows. Reader overhead is more or less constant and >>>>>>>>>>>>>>>>>>> is very >>>>>>>>>>>>>>>>>>> predictable. >>>>>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>>>>> > The main cost of this mode is that deletes must be >>>>>>>>>>>>>>>>>>> determined at write time which is expensive and can be more >>>>>>>>>>>>>>>>>>> difficult for >>>>>>>>>>>>>>>>>>> conflict resolution >>>>>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>>>>> > 2) Equality Deletes >>>>>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>>>>> > Writers write out reference to what values are deleted >>>>>>>>>>>>>>>>>>> (in a partition or globally). There can be an unlimited >>>>>>>>>>>>>>>>>>> number of equality >>>>>>>>>>>>>>>>>>> deletes and they all must be checked for every data file >>>>>>>>>>>>>>>>>>> that is read. The >>>>>>>>>>>>>>>>>>> cost of determining deleted rows is essentially given to >>>>>>>>>>>>>>>>>>> the reader. >>>>>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>>>>> > Conflicts almost never happen since data files are not >>>>>>>>>>>>>>>>>>> actually changed and there is almost no cost to the writer >>>>>>>>>>>>>>>>>>> to generate >>>>>>>>>>>>>>>>>>> these. Almost all costs related to equality deletes are >>>>>>>>>>>>>>>>>>> passed on to the >>>>>>>>>>>>>>>>>>> reader. >>>>>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>>>>> > Proposal: >>>>>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>>>>> > Equality deletes are, in my opinion, unsustainable and >>>>>>>>>>>>>>>>>>> we should work on deprecating and removing them from the >>>>>>>>>>>>>>>>>>> specification. At >>>>>>>>>>>>>>>>>>> this time, I know of only one engine (Apache Flink) which >>>>>>>>>>>>>>>>>>> produces these >>>>>>>>>>>>>>>>>>> deletes but almost all engines have implementations to read >>>>>>>>>>>>>>>>>>> them. The cost >>>>>>>>>>>>>>>>>>> of implementing equality deletes on the read path is >>>>>>>>>>>>>>>>>>> difficult and >>>>>>>>>>>>>>>>>>> unpredictable in terms of memory usage and compute >>>>>>>>>>>>>>>>>>> complexity. We’ve had >>>>>>>>>>>>>>>>>>> suggestions of implementing rocksdb inorder to handle ever >>>>>>>>>>>>>>>>>>> growing sets of >>>>>>>>>>>>>>>>>>> equality deletes which in my opinion shows that we are >>>>>>>>>>>>>>>>>>> going down the wrong >>>>>>>>>>>>>>>>>>> path. >>>>>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>>>>> > Outside of performance, Equality deletes are also >>>>>>>>>>>>>>>>>>> difficult to use in conjunction with many other features. >>>>>>>>>>>>>>>>>>> For example, any >>>>>>>>>>>>>>>>>>> features requiring CDC or Row lineage are basically >>>>>>>>>>>>>>>>>>> impossible when >>>>>>>>>>>>>>>>>>> equality deletes are in use. When Equality deletes are >>>>>>>>>>>>>>>>>>> present, the state >>>>>>>>>>>>>>>>>>> of the table can only be determined with a full scan making >>>>>>>>>>>>>>>>>>> it difficult to >>>>>>>>>>>>>>>>>>> update differential structures. This means materialized >>>>>>>>>>>>>>>>>>> views or indexes >>>>>>>>>>>>>>>>>>> need to essentially be fully rebuilt whenever an equality >>>>>>>>>>>>>>>>>>> delete is added >>>>>>>>>>>>>>>>>>> to the table. >>>>>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>>>>> > Equality deletes essentially remove complexity from >>>>>>>>>>>>>>>>>>> the write side but then add what I believe is an >>>>>>>>>>>>>>>>>>> unacceptable level of >>>>>>>>>>>>>>>>>>> complexity to the read side. >>>>>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>>>>> > Because of this I suggest we deprecate Equality >>>>>>>>>>>>>>>>>>> Deletes in V3 and slate them for full removal from the >>>>>>>>>>>>>>>>>>> Iceberg Spec in V4. >>>>>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>>>>> > I know this is a big change and compatibility breakage >>>>>>>>>>>>>>>>>>> so I would like to introduce this idea to the community and >>>>>>>>>>>>>>>>>>> solicit >>>>>>>>>>>>>>>>>>> feedback from all stakeholders. I am very flexible on this >>>>>>>>>>>>>>>>>>> issue and would >>>>>>>>>>>>>>>>>>> like to hear the best issues both for and against removal >>>>>>>>>>>>>>>>>>> of Equality >>>>>>>>>>>>>>>>>>> Deletes. >>>>>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>>>>> > Thanks everyone for your time, >>>>>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>>>>> > Russ Spitzer >>>>>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>> -- >>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>> *Jason Fine* >>>>>>>>>>>>>>>>>> Chief Software Architect >>>>>>>>>>>>>>>>>> ja...@upsolver.com | www.upsolver.com >>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>> >>>>>>>>>
