Re: Timestamp Watermark Assigner bpund question

2019-04-14 Thread Guowei Ma
sorry for missing a not. :( Whether the watermark, which is generated by the AssignerWithPunctuatedWatermarks/AssignerWithPeriodicWatermarks is send to the downstream is controlled by the framework. If an operator returns a watermark going back Flink would _*not_* send it to the downstream. Best,

Re: Timestamp Watermark Assigner bpund question

2019-04-14 Thread Guowei Ma
Hi, Vijay >>>Then the Operator progresses to the next Watermark as a starting point for events after event time reaches currWatermark ? AFAIK, the operator that generates watermark is called by the frame work. When the operator is called depends on the operator itself. For example the operator tha

Re: Timestamp Watermark Assigner bpund question

2019-04-10 Thread Vijay Balakrishnan
Hi Guowei, Thx for your reply. I am trying to understand the logic behind the Point 1 i.e current Watermark being currMaxTimestamp minus the bound. So, does this mean the Operator processing a task has a current Event time < current Watermark < currMaxTimestamp ??? Then the Operator progresses to t

Re: Timestamp Watermark Assigner bpund question

2019-04-09 Thread Guowei Ma
Hi, 1. From doc[1], A Watermark(t) declares that event time has reached time t in that stream, meaning that there should be no more elements from the stream with a timestamp t’ <= t (i.e. events with timestamps older or equal to the watermark). So I think it might be counterintuitive that generatin

Timestamp Watermark Assigner bpund question

2019-04-09 Thread Vijay Balakrishnan
Hi, I have created a TimestampAssigner as follows. I want to use monitoring.getEventTimestamp() with an Event Time processing and collected aggregated stats over time window intervals of 5 secs, 5 mins etc. Is this the right way to create the TimeWaterMarkAssigner with a bound ? I want to collect t