pause and resume flink stream job based on certain condition

2020-11-29 Thread Eleanore Jin
Hi experts, Here is my use case, it's a flink stateless streaming job for message validation. 1. read from a kafka topic 2. perform validation of message, which requires query external system 2a. the metadata from the external system will be cached in memory for 15minutes 2b. there i

Re: pause and resume flink stream job based on certain condition

2020-11-29 Thread Guowei Ma
Hi, Eleanore 1. AFAIK I think only the job could "pause" itself. For example the "query" external system could pause when the external system is down. 2. Maybe you could try the "iterate" and send the failed message back to retry if you use the DataStream api. Best, Guowei On Mon, Nov 30, 2020

Re: pause and resume flink stream job based on certain condition

2020-11-29 Thread Arvid Heise
Hi Eleanore, if the external system is down, you could simply fail the job after a given timeout (for example, using asyncIO). Then the job would restart using the restarting policies. If your state is rather small (and thus recovery time okay), you would pretty much get your desired behavior. Th