Re: Reactive mode and checkpointing

2022-04-13 Thread aryan m
Thanks for the pointers Alexander and David! We are exploring reactive mode on Flink 1.13 and my questions are merely hypothetical. Our streaming job consumes from Kafka, performs enrichment by querying external services and sinks to S3. Under backpressure or at random times, we observed one or t

Re: Reactive mode and checkpointing

2022-04-12 Thread David Morávek
Hi Aryan, this is an interesting thought. What kind of option do you have in mind? My take on this is that if checkpoint times out, it's pretty likely that the next one will timeout as well and the scheduler has no way of knowing that the next one would succeed. Also up-scaling might help to mitig

Re: Reactive mode and checkpointing

2022-04-12 Thread Alexander Preuß
Hello, There are no scheduler-specific options for checkpointing. You can however set `execution.checkpointing.tolerable-failed-checkpoints` to 0 to forbid checkpoint failures ( https://nightlies.apache.org/flink/flink-docs-release-1.14/docs/deployment/config/#execution-checkpointing-tolerable-fai