Re: Perform processing only when watermark updates, buffer data otherwise

2020-04-05 Thread Manas Kale
Hi Timo, Thanks for the information. On Thu, Apr 2, 2020 at 9:30 PM Timo Walther wrote: > Hi Manas, > > first of all, after assigning watermarks at the source level, usually > Flink operators make sure to handle the watermarks. > > In case of a `union()`, the subsequent operator will increment i

Re: Perform processing only when watermark updates, buffer data otherwise

2020-04-02 Thread Timo Walther
Hi Manas, first of all, after assigning watermarks at the source level, usually Flink operators make sure to handle the watermarks. In case of a `union()`, the subsequent operator will increment its internal event-time clock and emit a new watermark only if all input streams (and their paral

Re: Perform processing only when watermark updates, buffer data otherwise

2020-04-02 Thread Manas Kale
Also - What happens to watermarks after a union operation? Do I have to assignTimestampsAndWatermarks() again? I guess I will have to since multiple streams are being combined and Flink needs to know how to resolve individual watermarks. - What is the difference between union() and

Perform processing only when watermark updates, buffer data otherwise

2020-04-01 Thread Manas Kale
Hi, I want to perform some processing on events only when the watermark is updated. Otherwise, for all other events, I want to keep buffering them till the watermark arrives. The main motivation behind doing this is that I have several operators that emit events/messages to a downstream operator. S