Hi Jacob. Flink uses "notification" to let an operator callback the completion of a checkpoint. After gathering all checkpoint done messages from TMs, JM sends a "notify checkpoint completed" RPC to all TMs. Operators will handle this notification, where checkpoint success callbacks are invoked. For example, Kafka sources commit the current consuming offset. I think this doc ( https://flink.apache.org/2018/02/28/an-overview-of-end-to-end-exactly-once-processing-in-apache-flink-with-apache-kafka-too/) may be helpful.
You can override the `notifyCheckpointComlete()` to customize the behavior of handling checkpoint completion. Best regards Rui Xia On Fri, Mar 8, 2024 at 3:03 AM Jacob Rollings <jacobrolling...@gmail.com> wrote: > > Hello, > > I am implementing proof of concepts based Flink realtime streaming > solutions. > > I came across below lines in out-of-the-box Flink Kafka connector > documents. > > > > *https://nightlies.apache.org/flink/flink-docs-master/docs/connectors/datastream/kafka/* > <https://nightlies.apache.org/flink/flink-docs-master/docs/connectors/datastream/kafka/> > > *Consumer Offset Committing # > <https://nightlies.apache.org/flink/flink-docs-master/docs/connectors/datastream/kafka/#consumer-offset-committing>* > > *Kafka source commits the current consuming offset when checkpoints > are completed, for ensuring the consistency between Flink’s checkpoint > state and committed offsets on Kafka brokers*. > > > How is Flink able to control the callbacks from checkpointing? Is there a > way to override this into my implementations. I have multiple upstream > sources to connect to depending on the business model which are not Kafka. > Based on criticality of the system and publisher dependencies, we cannot > switch to Kafka for these. So I was hoping to do the same which kafka > connector is doing. > > > Cheers, > > JR >
