I just realized I did not open comments, fixed now. Any feedback or alternative ideas are more than welcome!
I'd say we should equally consider all incoming ideas at the moment and see what would work best. We haven't agreed on anything yet, so I'd encourage everyone to participate in the discussion. I will be off next week, will take another look once back. On 2023/10/12 01:20:50 Renjie Liu wrote: > Hi, Anton: > I've gone through the doc, and we are trying to solve the same problems of > position deletes, but with different approaches. It's quite interesting. > > On Thu, Oct 12, 2023 at 12:11 AM Anton Okolnychyi <aokolnyc...@apache.org> > wrote: > > > I tried to summarize notes from our previous discussions here: > > > > https://docs.google.com/document/d/1M4L6o-qnGRwGhbhkW8BnravoTwvCrJV8VvzVQDRJO5I/ > > > > I am going to iterate on the doc later today. > > > > On 2023/10/11 07:06:07 Renjie Liu wrote: > > > Hi, Russell: > > > > > > > > > > The main things I’m still interested are alternative approaches. I > > think > > > > that some of the work that Anton is working on have shown some > > different > > > > bottlenecks in applying delete files that I’m not sure are addressed by > > > > this proposal. > > > > > > > > > I'm also interested. Could you share some resources on the work that > > Anton > > > is working? I didn't notice that. > > > > > > For example, this proposal suggests doing a 1 to 1 (or 1 rowgroup to 1) > > > > delete file application in order to speed up planning. But this could > > as be > > > > done with a puffin file indexing delete files to data files. This would > > > > eliminate any planning cost while also allowing us to do more > > complicated > > > > things like mapping multiple data files to a single delete file as > > well as > > > > operate on a one to many data file to delete file approach. Doing this > > > > approach would mean we would need to change any existing metadata or > > > > introduce a new separate file type. > > > > > > > > > Yes, we can improve planning performance by embedding the mapping in a > > > puffin file. But I guess this may introduce other problems like > > conflicting > > > when doing commits? IIUC, puffin file is used as table level index or > > > statistics. > > > > > > I would also expect some POC experiments showing that the Spec is getting > > > > the benefit’s that are hypothesized. > > > > > > > > > I will conduct some poc experiments with actual data, but it may take > > some > > > time to implement it. > > > > > > The proposal I think also needs to address any possible limitations with > > > > this approach. They don’t all need to be solved but we should at least > > > > being exploring them. As a quick example, how does using single delete > > > > files interact with our commit logic? I would guess that a single > > delete > > > > file approach would make it more difficult to perform multiple deletes > > > > concurrently? > > > > > > > > > Good suggestion, I'm working on updating the doc to completement sketches > > > for dml operations. IIUC, for potential conflicts in performing multiple > > > deletes concurrently, you mean concurrent writes from different dml jobs? > > > If so, I think the current solution still has the same problem since this > > > is in fact conflicts from concurrent updates. But I do admit that the > > > deletion vector approach makes confliction easier since it's file level. > > > > > > > > > > > > On Tue, Oct 10, 2023 at 8:54 AM Russell Spitzer < > > russell.spit...@gmail.com> > > > wrote: > > > > > > > The main things I’m still interested are alternative approaches. I > > think > > > > that some of the work that Anton is working on have shown some > > different > > > > bottlenecks in applying delete files that I’m not sure are addressed by > > > > this proposal. > > > > > > > > For example, this proposal suggests doing a 1 to 1 (or 1 rowgroup to 1) > > > > delete file application in order to speed up planning. But this could > > as be > > > > done with a puffin file indexing delete files to data files. This would > > > > eliminate any planning cost while also allowing us to do more > > complicated > > > > things like mapping multiple data files to a single delete file as > > well as > > > > operate on a one to many data file to delete file approach. Doing this > > > > approach would mean we would need to change any existing metadata or > > > > introduce a new separate file type. > > > > > > > > I think basically for every “benefit” outlined we should think about if > > > > there is an alternative approach that would achieve the same benefit. > > Then > > > > we should analyze or whether or not the proposal is the best solution > > for > > > > that particular benefit and do some work to calculate what that benefit > > > > would be and what drawbacks there might be. > > > > > > > > I would also expect some POC experiments showing that the Spec is > > getting > > > > the benefit’s that are hypothesized. > > > > > > > > The proposal I think also needs to address any possible limitations > > with > > > > this approach. They don’t all need to be solved but we should at least > > > > being exploring them. As a quick example, how does using single delete > > > > files interact with our commit logic? I would guess that a single > > delete > > > > file approach would make it more difficult to perform multiple deletes > > > > concurrently? > > > > > > > > Sent from my iPad > > > > > > > > On Oct 8, 2023, at 9:22 PM, Renjie Liu <liurenjie2...@gmail.com> > > wrote: > > > > > > > > > > > > Hi, Ryan: > > > > Thanks for your reply. > > > > > > > > 1. What is the exact file format for these on disk that you're > > proposing? > > > >> Even if you're saying that it is what is produced by roaring bitmap, > > we > > > >> need more information. Is that a portable format? Do you wrap it at > > all in > > > >> the file to carry extra metadata? For example, the proposal says that > > a > > > >> starting position for a bitmap would be used. Where is that stored? > > > > > > > > > > > > Sorry for the confusion, by file format I mean roaring bitmap's file > > > > format < > > https://github.com/RoaringBitmap/RoaringFormatSpec#general-layout>. > > > > I checked that it has been implemented in several languages, such as > > java, > > > > go, rust, c. Metadata will be stored in manifest file as other entries > > such > > > > as datafile, deletion file. The starting position doesn't need to be > > stored > > > > since it's used by the file reader. I think your suggestion to provide > > an > > > > interface in design will make things clearer, and I will add it to the > > > > design doc. > > > > > > > > 2. How would DML operations work? Just a sketch would be great. I don't > > > >> think it is a good idea to leave the implications for DML fuzzy. > > > > > > > > > > > > I'll add sketches for other DML operations. > > > > > > > > 3. The comparison appears to be between rewriting data files and using > > > >> delete vectors. I think it needs to compare the existing delete file > > > >> formats to delete vectors so that we know why there is a benefit to > > doing > > > >> this beyond using the current positional delete files. The doc states > > that > > > >> there aren't measurements here, which I think we need. Otherwise, > > should we > > > >> just have a version of DML that produces one position delete per data > > file? > > > > > > > > > > > > I think deletion vector files are quite similar to position delete > > files, > > > > e.g. you can think of a deletion vector file as one position delete per > > > > data file. But this change brings new chances for optimization, and > > there > > > > is one section talking about it in the design doc. As with the > > > > measurements, I'll try to design some experiments for it. > > > > > > > > 4. I think this is missing some justification for how you're changing > > data > > > >> file metadata. > > > > > > > > > > > > I agree with your comment that if we associate one deletion vector > > with a > > > > data file, maybe it's better to extend the DataFile struct rather than > > > > introducing new entries. > > > > > > > > I'll update the doc to address the comments. > > > > > > > > > > > > > > > > > > > > > > > > > > > > On Mon, Oct 9, 2023 at 1:44 AM Ryan Blue <b...@tabular.io> wrote: > > > > > > > >> Thanks, Renjie. I went through and made some comments about what is > > still > > > >> not clear. Here's a summary: > > > >> > > > >> 1. What is the exact file format for these on disk that you're > > proposing? > > > >> Even if you're saying that it is what is produced by roaring bitmap, > > we > > > >> need more information. Is that a portable format? Do you wrap it at > > all in > > > >> the file to carry extra metadata? For example, the proposal says that > > a > > > >> starting position for a bitmap would be used. Where is that stored? > > > >> 2. How would DML operations work? Just a sketch would be great. I > > don't > > > >> think it is a good idea to leave the implications for DML fuzzy. > > > >> 3. The comparison appears to be between rewriting data files and using > > > >> delete vectors. I think it needs to compare the existing delete file > > > >> formats to delete vectors so that we know why there is a benefit to > > doing > > > >> this beyond using the current positional delete files. The doc states > > that > > > >> there aren't measurements here, which I think we need. Otherwise, > > should we > > > >> just have a version of DML that produces one position delete per data > > file? > > > >> 4. I think this is missing some justification for how you're changing > > > >> data file metadata. > > > >> > > > >> On Sat, Oct 7, 2023 at 4:49 AM Renjie Liu <liurenjie2...@gmail.com> > > > >> wrote: > > > >> > > > >>> Hi: > > > >>> I have addressed most comments in the document. I would like to ask > > > >>> what's the next step? Should we have a vote on this spec to reject > > it or we > > > >>> should go on with it? > > > >>> > > > >>> On Sat, Sep 30, 2023 at 11:20 PM Renjie Liu <liurenjie2...@gmail.com > > > > > > >>> wrote: > > > >>> > > > >>>> Hi: > > > >>>> Sorry for the late reply, I have been busy recently. I've updated > > the > > > >>>> design with more details about your questions, and here is a > > summary: > > > >>>> > > > >>>> > 1. Would there be only one delete vector per data file? > > > >>>> Yes. It's possible that we have multiple deletion vectors per very > > > >>>> large data file to further reduce write amplification, but I'm not > > sure if > > > >>>> it's over design. > > > >>>> > > > >>>> > 2. Would this require merge of existing vectors and new deletes at > > > >>>> write time? > > > >>>> Yes. Merging two bitmaps would be quite efficient. > > > >>>> > > > >>>> > 3. How would the data file for a vector be identified? > > > >>>> It will be stored in the manifest file. We will have one entry for > > > >>>> deletion file, and we add an extra field `data_file_path` for the > > > >>>> associated data file path. See Changes to spec > > > >>>> < > > https://docs.google.com/document/d/1FtPI0TUzMrPAFfWX_CA9NL6m6O1uNSxlpDsR-7xpPL0/edit#heading=h.p4vrosjzl14j> > > for > > > >>>> details, and Write process > > > >>>> < > > https://docs.google.com/document/d/1FtPI0TUzMrPAFfWX_CA9NL6m6O1uNSxlpDsR-7xpPL0/edit#heading=h.tft7a34rd2be> > > for > > > >>>> example. > > > >>>> > > > >>>> > 4. If multiple vectors are allowed, what is the plan for keeping > > the > > > >>>> number of delete vectors small? > > > >>>> I see multiple vectors per data file as an optimization for very > > large > > > >>>> data file, and I'm not sure if it's over design. > > > >>>> > > > >>>> > 5. Would we allow writing multiple delete vectors into the same > > file? > > > >>>> I don't want to do that. Merging delete vectors into one file have > > two > > > >>>> concerns: > > > >>>> > > > >>>> - Write amplification. > > > >>>> - It makes concurrent modification of data files difficult. > > > >>>> > > > >>>> > 6. How would we track which files are affected by a combined file > > of > > > >>>> delete vectors? > > > >>>> Sorry, I don't quite get your point. > > > >>>> > > > >>>> > 7. What are the details of the proposed file format? > > > >>>> I think roaring bitmap would be a good candidate, but other columnar > > > >>>> formats such as parquet, orc are also possible since they provided > > great > > > >>>> compression for boolean columns. I've mentioned it here > > > >>>> < > > https://docs.google.com/document/d/1FtPI0TUzMrPAFfWX_CA9NL6m6O1uNSxlpDsR-7xpPL0/edit#heading=h.nrhcjanzai0v > > > > > > >>>> > > > >>>> On Thu, Sep 21, 2023 at 4:53 PM Jean-Baptiste Onofré < > > j...@nanthrax.net> > > > >>>> wrote: > > > >>>> > > > >>>>> Hi Ryan, > > > >>>>> > > > >>>>> Thanks for the feedback. Unfortunately, I was not able to join the > > > >>>>> Iceberg community sync meeting yesterday, I promise I will join the > > > >>>>> next ones. > > > >>>>> > > > >>>>> I think the proposal is very interesting and also the > > > >>>>> discussion/comments in the document. I agree that some points > > should > > > >>>>> be discussed further. I propose to update the document with your > > > >>>>> points/questions. > > > >>>>> > > > >>>>> Thanks ! > > > >>>>> > > > >>>>> Regards > > > >>>>> JB > > > >>>>> > > > >>>>> On Thu, Sep 21, 2023 at 2:02 AM Ryan Blue <b...@tabular.io> wrote: > > > >>>>> > > > > >>>>> > Renjie, thanks for the proposal. > > > >>>>> > > > > >>>>> > We talked about this today in the Iceberg community sync and the > > > >>>>> general feedback was that we're excited work on this, but the > > proposal left > > > >>>>> a few areas unclear. There are a few decisions about how to manage > > the > > > >>>>> delete vectors that need to be added to the design. For example: > > > >>>>> > 1. Would there be only one delete vector per data file? > > > >>>>> > 2. Would this require merge of existing vectors and new deletes > > at > > > >>>>> write time? > > > >>>>> > 3. How would the data file for a vector be identified? > > > >>>>> > 4. If multiple vectors are allowed, what is the plan for keeping > > the > > > >>>>> number of delete vectors small? > > > >>>>> > 5. Would we allow writing multiple delete vectors into the same > > file? > > > >>>>> > 6. How would we track which files are affected by a combined > > file of > > > >>>>> delete vectors? > > > >>>>> > 7. What are the details of the proposed file format? > > > >>>>> > > > > >>>>> > In short, we just want to better understand how all this would > > work. > > > >>>>> > > > > >>>>> > Thanks! > > > >>>>> > > > > >>>>> > Ryan > > > >>>>> > > > > >>>>> > > > > >>>>> > On Mon, Sep 18, 2023 at 8:22 PM Renjie Liu < > > liurenjie2...@gmail.com> > > > >>>>> wrote: > > > >>>>> >> > > > >>>>> >> Hi, all: > > > >>>>> >> > > > >>>>> >> > > > >>>>> >> > > > >>>>> >> I have a proposal to introduce deletion vector file to reduce > > write > > > >>>>> amplification of iceberg table: > > > >>>>> >> > > > >>>>> >> > > > >>>>> > > https://docs.google.com/document/d/1FtPI0TUzMrPAFfWX_CA9NL6m6O1uNSxlpDsR-7xpPL0/edit?usp=sharing > > > >>>>> >> > > > >>>>> >> > > > >>>>> >> > > > >>>>> >> Welcome to comment, and looking forward to hear your advice. > > > >>>>> > > > > >>>>> > > > > >>>>> > > > > >>>>> > -- > > > >>>>> > Ryan Blue > > > >>>>> > Tabular > > > >>>>> > > > >>>> > > > >>>> > > > >>>> -- > > > >>>> Renjie Liu > > > >>>> Software Engineer, MVAD > > > >>>> > > > >>> > > > >>> > > > >>> -- > > > >>> Renjie Liu > > > >>> Software Engineer, MVAD > > > >>> > > > >> > > > >> > > > >> -- > > > >> Ryan Blue > > > >> Tabular > > > >> > > > > > > > > > > > > -- > > > > Renjie Liu > > > > Software Engineer, MVAD > > > > > > > > > > > > > > -- > > > Renjie Liu > > > Software Engineer, MVAD > > > > > > > > -- > Renjie Liu > Software Engineer, MVAD >
