Re: Per Partition Watermarking source idleness

2019-08-29 Thread Eduardo Winpenny Tejedor
Hey Prakhar, Sorry for taking so long to reply. One possible strategy is to advance the watermark after not receiving new messages for T milliseconds. In order to do this you must be fairly confident that you will not get messages delayed for longer than T milliseconds. To this end I've written my

Re: Per Partition Watermarking source idleness

2019-08-25 Thread Prakhar Mathur
Hi, Thanks for the response. Can you point me to some examples of such strategy? On Sat, Aug 24, 2019, 06:01 Eduardo Winpenny Tejedor < eduardo.winpe...@gmail.com> wrote: > Hi Prakhar, > > Everything is probably working as expected, if a partition does not > receive any messages then the waterma

Re: Per Partition Watermarking source idleness

2019-08-23 Thread Eduardo Winpenny Tejedor
Hi Prakhar, Everything is probably working as expected, if a partition does not receive any messages then the watermark of the operator does not advance (as it is the minimum across all partitions). You'll need to define a strategy for the watermark to advance even when no messages are received f