Actually, how are you committing offsets? Are you using the old (zookeeperconsumerconnector) or new KafkaConsumer?
It is true that the current APIs don't return any result, but it would help to check if anything is getting into the offsets topic - unless you are seeing errors in the logs, the offset commit should succeed (if you are indeed explicitly committing offsets). Thanks, Joel On Tue, Jul 14, 2015 at 12:19:01PM -0400, Vadim Bobrov wrote: > Thanks, Joel, I will but regardless of my findings the basic problem will > still be there: there is no guarantee that the offsets will be committed > after commitOffsets. Because commitOffsets does not return its exit status, > nor does it block as I understand until offsets are committed. In other > words, there is no way to know that it has, in fact, commited the offsets > > or am I missing something? And then another question - why does it seem to > depend on the number of consumed messages? > > On Tue, Jul 14, 2015 at 11:36 AM, Joel Koshy <jjkosh...@gmail.com> wrote: > > > Can you take a look at the kafka commit rate mbean on your consumer? > > Also, can you consume the offsets topic while you are committing > > offsets and see if/what offsets are getting committed? > > (http://www.slideshare.net/jjkoshy/offset-management-in-kafka/32) > > > > Thanks, > > > > Joel > > > > On Tue, Jul 14, 2015 at 11:12:03AM -0400, Vadim Bobrov wrote: > > > I am trying to replace ActiveMQ with Kafka in our environment however I > > > have encountered a strange problem that basically prevents from using > > Kafka > > > in production. The problem is that sometimes the offsets are not > > committed. > > > > > > I am using Kafka 0.8.2.1, offset storage = kafka, high level consumer, > > > auto-commit = off. Every N messages I issue commitOffsets(). Now here is > > > the problem - if N is below a certain number (180 000 for me) it works > > and > > > the offset is moving. If N is 180 000 or more the offset is not updated > > > after commitOffsets > > > > > > I am looking at offsets using kafka-run-class.sh > > > kafka.tools.ConsumerOffsetChecker > > > Any help? > > > >
