Thanks, Joel, I will but regardless of my findings the basic problem will still be there: there is no guarantee that the offsets will be committed after commitOffsets. Because commitOffsets does not return its exit status, nor does it block as I understand until offsets are committed. In other words, there is no way to know that it has, in fact, commited the offsets
or am I missing something? And then another question - why does it seem to depend on the number of consumed messages? On Tue, Jul 14, 2015 at 11:36 AM, Joel Koshy <jjkosh...@gmail.com> wrote: > Can you take a look at the kafka commit rate mbean on your consumer? > Also, can you consume the offsets topic while you are committing > offsets and see if/what offsets are getting committed? > (http://www.slideshare.net/jjkoshy/offset-management-in-kafka/32) > > Thanks, > > Joel > > On Tue, Jul 14, 2015 at 11:12:03AM -0400, Vadim Bobrov wrote: > > I am trying to replace ActiveMQ with Kafka in our environment however I > > have encountered a strange problem that basically prevents from using > Kafka > > in production. The problem is that sometimes the offsets are not > committed. > > > > I am using Kafka 0.8.2.1, offset storage = kafka, high level consumer, > > auto-commit = off. Every N messages I issue commitOffsets(). Now here is > > the problem - if N is below a certain number (180 000 for me) it works > and > > the offset is moving. If N is 180 000 or more the offset is not updated > > after commitOffsets > > > > I am looking at offsets using kafka-run-class.sh > > kafka.tools.ConsumerOffsetChecker > > Any help? > >
