Re: Dump snapshot of big table in real time using StreamingFileSink

2019-01-18 Thread Cristian C
ay, maybe Kostas can add more since he wrote the StreamingFileSink >> code. I've cc'd him directly. >> >> -Jamie >> >> >> On Fri, Jan 18, 2019 at 9:44 AM Cristian C >> wrote: >> >>> Well, the problem is that, conceptually, the way I'

Re: Dump snapshot of big table in real time using StreamingFileSink

2019-01-18 Thread Cristian C
Well, the problem is that, conceptually, the way I'm trying to approach this is ok. But in practice, it has some edge cases. So back to my original premise: if you both, trigger and checkpoint happen around the same time, there is a chance that the streaming file sink rolls the bucket BEFORE it ha