Can you take a look at the kafka commit rate mbean on your consumer? Also, can you consume the offsets topic while you are committing offsets and see if/what offsets are getting committed? (http://www.slideshare.net/jjkoshy/offset-management-in-kafka/32)
Thanks, Joel On Tue, Jul 14, 2015 at 11:12:03AM -0400, Vadim Bobrov wrote: > I am trying to replace ActiveMQ with Kafka in our environment however I > have encountered a strange problem that basically prevents from using Kafka > in production. The problem is that sometimes the offsets are not committed. > > I am using Kafka 0.8.2.1, offset storage = kafka, high level consumer, > auto-commit = off. Every N messages I issue commitOffsets(). Now here is > the problem - if N is below a certain number (180 000 for me) it works and > the offset is moving. If N is 180 000 or more the offset is not updated > after commitOffsets > > I am looking at offsets using kafka-run-class.sh > kafka.tools.ConsumerOffsetChecker > Any help?
