Re: Time-stamp of output from a window operator

2017-12-22 Thread Eron Wright
That is correct, the window operator happily accepts late elements into the applicable window(s). Though the element is late (with respect to the current watermark) it might still be included in the main firing, or provoke a late firing. Elements occurring downstream of an operator that uses late

Re: Time-stamp of output from a window operator

2017-12-22 Thread Jagadish
Thanks for the super-helpful reply! Would the late-firing from the "hourly" window be considered an on-time arrival for the subsequent "daily" window operator? Since, its windowing duration is much longer (1 day), I'd expect the "on-time/default" result emitted from the "daily" window to include

Re: Time-stamp of output from a window operator

2017-12-22 Thread Eron Wright
Each event in the stream has an associated timestamp as metadata. A timestamp assigner simply extracts the timestamp from the user object for that purpose. There is no per-operator watermark, but with `assignTimestampsAndWatermarks` you may insert an operator that overrides upstream watermarks.

Time-stamp of output from a window operator

2017-12-20 Thread Jagadish Venkatraman
Hey Flink developers, I'm trying to understand the behavior of timestamp assignments in Apache Flink. Here's my current understanding: 1. A *TimestampAssigner* returns the timestamp of each incoming message, while a *WatermarkAssigner* emits the watermark for a source. (If per-operator watermark