Hi, I have a question about the at-least-once guarantees for Kafka producers when checkpointing is disabled. In our data pipeline we have a Flink job on an unlimited stream that originally, we had checkpoints turned on. Further this job is cancelled with a savepoint once a day to do some data pre and post-processing for the next day, afterwards this job is restarted from the savepoint.
The issue we have is that we want to turn off checkpointing, since it does not give us much value and only creates extra IO. When this is done this message<https://github.com/apache/flink/blob/97bfd049951f8d52a2e0aed14265074c4255ead0/flink-connectors/flink-connector-kafka/src/main/java/org/apache/flink/streaming/connectors/kafka/FlinkKafkaProducerBase.java#L263> shows up: "Flushing on checkpoint is enabled, but checkpointing is not enabled. Disabling flushing." This prompted us to investigate, and it seems that if you have checkpointing disabled, there are no at-least-once guarantees. <https://github.com/apache/flink/blob/97bfd049951f8d52a2e0aed14265074c4255ead0/flink-connectors/flink-connector-kafka/src/main/java/org/apache/flink/streaming/connectors/kafka/FlinkKafkaProducerBase.java#L65> What about if you have no checkpointing, but you make savepoints that you restore from yourself? Savepoints are the same thing as checkpoints in the code. The flink producer makes it impossible to turn on flushing and have checkpointing disabled. I can see why this is the case as there is some extra synchronization overhead related to the flushing flag being on. Is there a way to have checkpointing disabled and have at least once guarantees on savepoints? The only thing I can think about is have checkpoints enabled with some very high periodicity so that they are never(almost) triggered. But this is a hack. Tomas Witzany
