Thanks Theo ! From: "theo.diefent...@scoop-software.de" <theo.diefent...@scoop-software.de> Date: Thursday, 13 February 2020 at 12:13 AM To: "Hegde, Mahendra" <mahendra.he...@arity.com>, "user@flink.apache.org" <user@flink.apache.org> Subject: [External] AW: How Flink Kafka Consumer works when it restarts
Hi Mahendra, Flink will regularly create checkpoints or manually triggered savepoints. This is data managed and stored by Flink and that data also contains the kafka offsets. When restarting, you can configure to restart from the last checkpoint and or savepoint. You can additionally configure Flink to commit the offsets to kafka, again, on checkpoint only. You can then configure Flink to restart from the committed offset, if you don't let Flink restart from an existing checkpoint or savepoint, where it would first search in to retore the offsets. Having the offsets loaded either from checkpoint, savepoint or kafka, it will directly communicate with Kafka and ask kafka to poll messages starting from those offsets. Best regards Theo Von meinem Huawei-Telefon gesendet -------- Ursprüngliche Nachricht -------- Von: "Hegde, Mahendra" <mahendra.he...@arity.com> Datum: Mi., 12. Feb. 2020, 17:50 An: user@flink.apache.org Betreff: How Flink Kafka Consumer works when it restarts Hi All, I am bit confused on Flink kafka consumer working. I read that Flink stores the kafka message offset in checkpoint and uses it in case if it restarts. Question is when exactly Flink is committing about successful consumption confirmation to kafka broker? And when Flink job restarts will it send last offset which is available in checkpoint to kafka broker to start consuming from that point ? Or Kafka broker will resume based on last committed offset information available? (I mean who manages the actual offset here, Kafka broker or the Flink client) Thanks Mahendra
