I created #7449 and added it to our 1.3.0 milestone to make sure it gets 
delivered.

- Anton

> On Apr 26, 2023, at 1:59 PM, Steven Wu <stevenz...@gmail.com> wrote:
> 
> > Will a clock skew cause any issues w.r.t. relying on the snapshot commit 
> > time? I think we allow a mismatch up to a minute in TableMetadata.
> 
> This is probably not a problem. typically the max allowed misalignment is 
> much longer than 1 minute.
> 
> > We also planned to expose file sequence number (different from data 
> > sequence number). I believe you could lookup snapshot using that info.
> 
> That would work too. 
> 
> 
> On Wed, Apr 26, 2023 at 1:10 PM Jack Ye <yezhao...@gmail.com 
> <mailto:yezhao...@gmail.com>> wrote:
> +1 for using file sequence number. This work has been discussed for a long 
> time but never got picked up, would be great if someone can drive it to 
> completion.
> 
> -Jack
> 
> On Wed, Apr 26, 2023 at 12:03 PM Anton Okolnychyi 
> <aokolnyc...@apple.com.invalid> wrote:
> Will a clock skew cause any issues w.r.t. relying on the snapshot commit 
> time? I think we allow a mismatch up to a minute in TableMetadata.
> 
> We also planned to expose file sequence number (different from data sequence 
> number). I believe you could lookup snapshot using that info.
> 
> https://iceberg.apache.org/spec/#manifest-entry-fields 
> <https://iceberg.apache.org/spec/#manifest-entry-fields>
> 
> - Anton
> 
>> On Apr 26, 2023, at 11:52 AM, Steven Wu <stevenz...@gmail.com 
>> <mailto:stevenz...@gmail.com>> wrote:
>> 
>> piggyback on this thread since we are discussing exposing more metadata in 
>> ContentFile or FileScanTask. Flink source watermark alignment can 
>> potentially leverage the snapshot timestamp (when data files are 
>> committed/appended to the table).  Is it reasonable to expose some snapshot 
>> metadata in the FileScanTask?
>> 
>> This can help the Flink job to ensure two (or more) sources are proceeding 
>> at similar paces. Some use cases may require column stats (min-max values) 
>> for the watermark alignment. Some use cases can leverage snapshot timestamps 
>> for the alignment purpose.
>> 
>> On Wed, Apr 26, 2023 at 11:15 AM Anton Okolnychyi 
>> <aokolnyc...@apple.com.invalid <mailto:aokolnyc...@apple.com.invalid>> wrote:
>> My initial thinking is that exposing sequence numbers on ContentFile is 
>> preferable (we would get it for free in scan tasks). That said, I’ll need to 
>> see how complicated the implementation would be. Exposing it on 
>> ContentScanTask is a viable alternative. However, we already have a 
>> precedent for assigning specId in InheritableMetadata.
>> 
>> - Anton
>> 
>>> On Apr 26, 2023, at 10:41 AM, Ryan Blue <b...@tabular.io 
>>> <mailto:b...@tabular.io>> wrote:
>>> 
>>> Exposing sequence number makes sense for use cases like this. I also like 
>>> the idea of exposing it through FileScanTask. That might be easier than 
>>> trying to add it to ContentFile.
>>> 
>>> Anton, what do you think about adding it to FileScanTask?
>>> 
>>> On Wed, Apr 26, 2023 at 7:50 AM Anton Okolnychyi 
>>> <aokolnyc...@apple.com.invalid <mailto:aokolnyc...@apple.com.invalid>> 
>>> wrote:
>>> It is actually my bad not following up on that after #5913 and #6002. I’ll 
>>> take a look at #5760 referenced below by the end of this week. 
>>> 
>>> The plan was to expose sequence numbers on ContentFile. It is needed in a 
>>> number of use cases.
>>> 
>>> - Anton
>>> 
>>>> On Apr 26, 2023, at 4:55 AM, Gabor Kaszab <gaborkas...@apache.org 
>>>> <mailto:gaborkas...@apache.org>> wrote:
>>>> 
>>>> Hey Iceberg Community,
>>>> 
>>>> I know there has been a discussion previously about exposing the sequence 
>>>> number on a ContentFile level, but if I'm not mistaken that conversation 
>>>> didn't end with a consensus. I found some relevant PRs that has been open 
>>>> for a while:
>>>> https://github.com/apache/iceberg/pull/5760 
>>>> <https://github.com/apache/iceberg/pull/5760>
>>>> https://github.com/apache/iceberg/pull/4769 
>>>> <https://github.com/apache/iceberg/pull/4769> (merged into the above PR)
>>>> 
>>>> The reason I bring this topic up is that we started investigating recently 
>>>> how to add read support for equality deletes to Impala. Apparently, 
>>>> implementation-wise we could save a lot of hassle if sequence numbers were 
>>>> exposed on a file level through the API, preferably somewhere around 
>>>> calling planFiles(). We could then have a virtual 'SEQUENCE_NUMBER' when 
>>>> scanning the data and delete files (separate scanners) and could easily 
>>>> filter the rows in the JOIN node that joins the rows from the data files 
>>>> with the ones from the delete files. (wouldn't go into more depth atm)
>>>> 
>>>> With this mail I'd like to revive this conversation with the hope of 
>>>> eventually coming to a solution that satisfies all participants. I've been 
>>>> thinking of implementation choices we have to somehow provide sequence 
>>>> numbers for the files:
>>>> - Extending ContentFile with sequence number: I checked the above PRs and 
>>>> IIUC the issue with this approach is that ContentFile is meant to be 
>>>> immutable and by the time they are created we don't have sequence numbers 
>>>> to populate the ContentFile object.
>>>> - Extend FileScanTask with the file-level sequence numbers so after 
>>>> calling planFiles() we could retrieve these numbers via a new API call on 
>>>> the FileScanTask.
>>>> 
>>>> There might be many other ways to implement this and I'd love to hear what 
>>>> people think and would be great to find a way that would help us out on 
>>>> Impala.
>>>> 
>>>> Cheers,
>>>> Gabor
>>>> 
>>>> 
>>> 
>>> 
>>> 
>>> -- 
>>> Ryan Blue
>>> Tabular
>> 
> 

Reply via email to