Re: Watermarks in Event Time windowing

2018-09-14 Thread David Anderson
To clarify one thing: keep in mind that Flink does not support per-key watermarks. Watermarks are typically generated per-source, or in the case of kafka, can be per-partition. An idle source (or in the case of kafka, an idle partition) can prevent an event-time window from being triggered, but you

Re: Watermarks in Event Time windowing

2018-09-13 Thread Taher Koitawala
Yes in many cases what we have faced that let's say in a keyed stream an element of a specific key comes in which triggers a new window. If a corresponding elements of the same key does not arrive a new watermark is not generated for the window to purge. Then we faced issues with flink keeping reco

Re: Watermarks in Event Time windowing

2018-09-13 Thread vino yang
Hi Taher, For some questions, I suggest you read the documentation related to Flink EventTime first, for example [1] About this question: What happens if the watermark is same as the timestamp? Here "timestamp", do you mean the current timestamp of Processing time? If that's the best, it's an id