Re: Watermark doesn't progress after job restore from savepoint

2021-03-04 Thread Guowei Ma
Hi, I think you could try implementing the `CheckpointedFunction` interface and `FunctionInitializationContext.isRestored` is an indicator for that. BTW: I am not very sure your scenarios but maybe you could try to set idleness configurations [1] [1] https://ci.apache.org/projects/flink/flink-doc

Re: Watermark doesn't progress after job restore from savepoint

2021-03-04 Thread bat man
Thanks Piotr. Got it. Had to push the static rules to the kafka queue as it had expired and got archived from the topic. Post this the pipeline resumed. To your suggestion on implementing an operator that remembers the watermark, is there any indicator that the job has been resumed which I can use

Re: Watermark doesn't progress after job restore from savepoint

2021-03-04 Thread Piotr Nowojski
Hi Hemant, State of the latest seen watermarks is not persisted in the operators. Currently DataStream API assumes that after recovery watermarks are going to be re-emitted sooner or later. What probably happens is that one of your sources has emitted watermarks (maybe some very high one or even `

Watermark doesn't progress after job restore from savepoint

2021-03-04 Thread bat man
Hi All, I have a job where my source is kafka. Stream1 is partition the data on dynamic key, join the data with static rules(source kafka).I use KeyedCoProcessFunction to join the Steam1 with Stream2(source kafka). All works fine in a normal run. For changing the watermark generation interval I s