Re: Time-sliced incremental scan

2022-02-07 Thread Peidian Li
the rolled back snapshot if the >>>> timestamp is in the interval when it was the table’s current state. >>>> >>>> We will need to decide whether VERSIONS BETWEEN ... uses table history >>>> or snapshot creation timestamps. If it uses table history, there

Re: Time-sliced incremental scan

2022-02-02 Thread Ryan Blue
uld fail. If it >>> uses snapshot creation timestamps, then you’d be able to select intervals >>> that never really existed, like between commits in a transaction. >>> >>> So there are major issues with timestamps. We would want to make it >>> clear that

Re: Time-sliced incremental scan

2022-01-20 Thread Walaa Eldin Moustafa
may fail). That said, it’s a really convenient feature and >> I would support adding both VERSIONS BETWEEN with timestamp and snapshot >> ID. >> >> Ryan >> >> On Thu, Jan 6, 2022 at 9:52 PM Walaa Eldin Moustafa < >> wa.moust...@gmail.com> wrote: >&g

Re: Time-sliced incremental scan

2022-01-08 Thread Kyle Bendickson
the issue from that thread) and may not reflect the actual > table state (or may fail). That said, it’s a really convenient feature and > I would support adding both VERSIONS BETWEEN with timestamp and snapshot > ID. > > Ryan > > On Thu, Jan 6, 2022 at 9:52 PM Walaa Eldin Moustafa &g

Re: Time-sliced incremental scan

2022-01-07 Thread Ryan Blue
both VERSIONS BETWEEN with timestamp and snapshot ID. Ryan On Thu, Jan 6, 2022 at 9:52 PM Walaa Eldin Moustafa wrote: > Hi Iceberg devs, > > We have been considering the problem of Time-sliced incremental scan > (i.e., reading data that is committed between two timestamps), and I ran

Time-sliced incremental scan

2022-01-06 Thread Walaa Eldin Moustafa
Hi Iceberg devs, We have been considering the problem of Time-sliced incremental scan (i.e., reading data that is committed between two timestamps), and I ran into this thread [1] in the Iceberg dev mailing list. The summary of the thread is that incremental scan should leverage snapshot IDs as