Re: Parallelism, registerEventTimeTimer and watermark problem

2017-10-20 Thread Aljoscha Krettek
Hi Fritz, If the watermark is not updating this usually means that one of the input partitions (if you're using Kafka) is not carrying data. In that case, the watermark/timestamp assigner will have no data on which to base an updated watermark. For such use cases I recently implemented a specia

Re: Parallelism, registerEventTimeTimer and watermark problem

2017-10-17 Thread Fritz Budiyanto
Sorry, missing copy paste for the exception thrown: 10/17/2017 20:21:30 dropDetection -> (aggFlowDropDetectPrintln -> Sink: Unnamed, aggFlowDropDetectPrintln -> Sink: Unnamed, Sink: kafkaSink)(3/4) switched to CANCELED 20:21:30,244 INFO org.apache.flink.runtime.executiongraph.ExecutionGrap

Parallelism, registerEventTimeTimer and watermark problem

2017-10-17 Thread Fritz Budiyanto
Hi All, If I have high parallelism and use processFunction to registerEventTimeTimer, the timer never gets fired. After debugging, I found out the watermark isn't updated because I have keyBy right after assignTimestampsAndWatermarks. And if I set assignTimestampsAndWatermarks right after the ke