Re: TimerService/Watermarks and Checkpoints

2018-06-01 Thread Narayanan Arunachalam
ate data per minute while the job catches up > until it reads data at the rate at which it is produced into Kafka. > > Would that explain your observations? > > Best, Fabian > > 2018-05-30 23:56 GMT+02:00 Narayanan Arunachalam < > narayanan.arunacha...@gmail.com>: > &g

Re: TimerService/Watermarks and Checkpoints

2018-05-30 Thread Narayanan Arunachalam
e the job is restarted from a previous checkpoint, it is reset to >>> Long.MIN_VALUE. I can see it a bit tricky to cover it into the checkpoint, >>> especially when we need to support rescaling(it seems not like a purely >>> keyed or a operate state), maybe @Stefan

Re: TimerService/Watermarks and Checkpoints

2018-05-30 Thread Narayanan Arunachalam
it tricky to cover it into the checkpoint, > especially when we need to support rescaling(it seems not like a purely > keyed or a operate state), maybe @Stefan or @Aljoscha could give you more > useful information about why it wasn't covered by the checkpoint. > > Best, Sihua > >

TimerService/Watermarks and Checkpoints

2018-05-29 Thread Narayanan Arunachalam
Hi, Is it possible the watermark in TimerService not getting reset when a job is restarted from a previous checkpoint? I would expect the watermark in a TimerService also to go back in time. I have the following ProcessFunction implementation. override def processElement( e: TraceEvent,