Re: Checkpointing Event Time Watermarks

2018-03-02 Thread Fabian Hueske
Hi Vijay, I think the easiest solution would be to inject a ProcessFunction after the window operator. The ProcessFunction has access to the current watermark via its Context object and can store it in union operator state. In case of a failure, the ProcessFunction recovers the watermark from its

Re: Checkpointing Event Time Watermarks

2018-03-02 Thread vijay kansal
Hi Xingcan We are receiving events from a no. of independent data sources and hence, data arriving into our Flink topology (via Kafka) would be out of order. We are creating 1-min event time windows in our Flink topology and generating event time watermarks as (current event time - some threshold

Re: Checkpointing Event Time Watermarks

2018-02-27 Thread Xingcan Cui
Hi Vijay, normally, maybe there’s no need to checkpoint the event times / watermarks since they are automatically generated based on the records. What’s your intention? Best, Xingcan > On 27 Feb 2018, at 8:50 PM, vijay kansal wrote: > > Hi All > > Is there a way to checkpoint event time wat

Checkpointing Event Time Watermarks

2018-02-27 Thread vijay kansal
Hi All Is there a way to checkpoint event time watermarks in Flink ? I tries searching for this, but could not figure out... Vijay Kansal Software Development Engineer LimeRoad