Hi Prabhu, cc Stephan, Robert, I was having similar issues where flink Kafka 09 consumer was not committing offsets to kafka. After digging into JobManager logs, I found that checkpoints were getting expired before getting completed and hence "checkpoint completed" message was being ignored.
I increased the checkpoint interval from default 10 mins to 30 mins to verify, and then checkpoints were getting finished way before timeout (~12 mins), and then consumer was correctly updating offsets in kafka. This seems to be working for us at the moment, and also note this scenarios normally happens at the start of the job and the consumer group already has some decent lag. So, you might wanna try increasing checkpoint timeouts and check if that solves the issue. You should look for following in the jobmanager logs [Checkpoint Timer] org.apache.flink.runtime.checkpoint.CheckpointCoordinator - Checkpoint 37 expired before completing. [Checkpoint Timer] org.apache.flink.runtime.checkpoint.CheckpointCoordinator - Triggering checkpoint 38 @ 1474313373634 [Checkpoint Timer] org.apache.flink.runtime.checkpoint.CheckpointCoordinator - Checkpoint 38 expired before completing. [Checkpoint Timer] org.apache.flink.runtime.checkpoint.CheckpointCoordinator - Triggering checkpoint 39 @ 1474313973640 -- Ankit -- View this message in context: http://apache-flink-user-mailing-list-archive.2336050.n4.nabble.com/Flink-Kafka-Consumer-Behaviour-tp8257p9300.html Sent from the Apache Flink User Mailing List archive. mailing list archive at Nabble.com.
