How about 9AM PDT on Friday, 5 July then? On Wed, Jul 3, 2019 at 10:55 AM Owen O'Malley <owen.omal...@gmail.com> wrote:
> I'd like to call in, but I'm out Thursday. Friday would work except 11am > to 1pm pdt. > > .. Owen > > On Wed, Jul 3, 2019 at 10:42 AM Ryan Blue <rb...@netflix.com.invalid> > wrote: > >> I'm available Thursday and Friday this week as well, but it's a holiday >> in the US so some people may be out. If there are no objections from anyone >> that would like to attend, then I'm up for that. >> >> On Wed, Jul 3, 2019 at 10:40 AM Anton Okolnychyi <aokolnyc...@apple.com> >> wrote: >> >>> I apologize for the delay on my side. I’ll still have to go through the >>> last emails. I am available on Thursday/Friday this week and would be great >>> to sync. >>> >>> Thanks, >>> Anton >>> >>> On 3 Jul 2019, at 01:29, Ryan Blue <rb...@netflix.com.INVALID> wrote: >>> >>> Sorry I didn't get back to this thread last week. Let's try to have a >>> video call to sync up on this next week. What days would work for everyone? >>> >>> rb >>> >>> On Fri, Jun 21, 2019 at 9:06 AM Erik Wright <erik.wri...@shopify.com> >>> wrote: >>> >>>> With regards to operation values. Currently they are: >>>> >>>> - append: data files were added and no files were removed. >>>> - replace: data files were rewritten with the same data; i.e., >>>> compaction, changing the data file format, or relocating data files. >>>> - overwrite: data files were deleted and added in a logical >>>> overwrite operation. >>>> - delete: data files were removed and their contents logically >>>> deleted. >>>> >>>> If deletion files (with or without data files) are appended to the >>>> dataset, will we consider that an `append` operation? If so, if deletion >>>> and/or data files are appended, and whole files are also deleted, will we >>>> consider that an `overwrite`? >>>> >>>> Given that the only apparent purpose of the operation field is to >>>> optimize snapshot expiration the above seems to meet its needs. An >>>> incremental reader can also skip `replace` snapshots but no others. Once it >>>> decides to read a snapshot I don't think there's any difference in how it >>>> processes the data for append/overwrite/delete cases. >>>> >>>> On Thu, Jun 20, 2019 at 8:55 PM Ryan Blue <rb...@netflix.com> wrote: >>>> >>>>> I don’t see that we need [sequence numbers] for file/offset-deletes, >>>>> since they apply to a specific file. They’re not harmful, but the don’t >>>>> seem relevant. >>>>> >>>>> These delete files will probably contain a path and an offset and >>>>> could contain deletes for multiple files. In that case, the sequence >>>>> number >>>>> can be used to eliminate delete files that don’t need to be applied to a >>>>> particular data file, just like the column equality deletes. Likewise, it >>>>> can be used to drop the delete files when there are no data files with an >>>>> older sequence number. >>>>> >>>>> I don’t understand the purpose of the min sequence number, nor what >>>>> the “min data seq” is. >>>>> >>>>> Min sequence number would be used for pruning delete files without >>>>> reading all the manifests to find out if there are old data files. If no >>>>> manifest with data for a partition contains a file older than some >>>>> sequence >>>>> number N, then any delete file with a sequence number < N can be removed. >>>>> >>>> OK, so the minimum sequence number is an attribute of manifest files. >>>> Sounds good. It can likely permit us to optimize compaction operations as >>>> well (i.e., you can easily limit the operation to a subset of manifest >>>> files as long as they are the oldest ones). >>>> >>>> >>>>> The “min data seq” is the minimum sequence number of a data file. That >>>>> seems like what we actually want for the pruning I described above. >>>>> >>>> I would expect a data file (appended rows or deletions by column value) >>>> to have a single sequence number that applies to the whole file. Even a >>>> delete-by-file-and-offset file can do with only a single sequence number >>>> (which must be larger than the sequence numbers of all deleted files). Why >>>> do we need a "minimum" data sequence per file? >>>> >>>>> Off the top of my head [supporting non-key delete] requires adding >>>>> additional information to the manifest file, indicating the columns that >>>>> are used for the deletion. Only equality would be supported; if multiple >>>>> columns were used, they would be combined with boolean-and. I don’t see >>>>> anything too tricky about it. >>>>> >>>>> Yes, exactly. I actually phrased it wrong initially. I think it would >>>>> be simple to extend the equality deletes to do this. We just need a way to >>>>> have global scope, not just partition scope. >>>>> >>>> I don't think anything special needs to be done with regards to >>>> scoping/partitioning of delete files. When scanning one or more data files, >>>> one must also consider any and all deletion files that could apply to them. >>>> The only way to prune deletion files from consideration is: >>>> >>>> 1. All of your data files have at least one partition column in >>>> common. >>>> 2. The deletion file is also partitioned on that column (at least). >>>> 3. The value sets of the data files do not overlap the value sets >>>> of the deletion files in that column. >>>> >>>> So given a dataset of sessions that is partitioned by device form >>>> factor and date, for example, you could have a delete (user_id=9876) in a >>>> deletion file that is not partitioned. And it would be "in scope" for all >>>> of those data files. >>>> >>>> If you had the same dataset partitioned by hash(user_id) and your >>>> deletes were _also_ partitioned by hash(user_id) you would be able to prune >>>> those deletes while scanning the sessions. >>>> >>>>> If we add this on a per-deletion file basis it is not clear if there >>>>> is any relevance in preserving the concept of a unique row ID. >>>>> >>>>> Agreed. That’s why I’ve been steering us away from the debate about >>>>> whether keys are unique or not. Either way, a natural key delete must >>>>> delete all of the records it matches. >>>>> >>>>> I would assume that the maximum sequence number should appear in the >>>>> table metadata >>>>> >>>>> Agreed. >>>>> >>>>> [W]ould you make it optional to assign a sequence number to a >>>>> snapshot? “Replace” snapshots would not need one. >>>>> >>>>> The only requirement is that it is monotonically increasing. If one >>>>> isn’t used, we don’t have to increment. I’d say it is up to the >>>>> implementation to decide. I would probably increment it every time to >>>>> avoid >>>>> errors. >>>>> -- >>>>> Ryan Blue >>>>> Software Engineer >>>>> Netflix >>>>> >>>> >>> >>> -- >>> Ryan Blue >>> Software Engineer >>> Netflix >>> >>> >>> >> >> -- >> Ryan Blue >> Software Engineer >> Netflix >> > -- Ryan Blue Software Engineer Netflix
