I am using ZookeeperConsumerConnector actually I set up a consumer for __consumer_offsets the way you suggested and now I cannot reproduce the situation any longer. Offsets are committed every time.
On Tue, Jul 14, 2015 at 1:49 PM, Joel Koshy <jjkosh...@gmail.com> wrote: > Actually, how are you committing offsets? Are you using the old > (zookeeperconsumerconnector) or new KafkaConsumer? > > It is true that the current APIs don't return any result, but it would > help to check if anything is getting into the offsets topic - unless > you are seeing errors in the logs, the offset commit should succeed > (if you are indeed explicitly committing offsets). > > Thanks, > > Joel > > On Tue, Jul 14, 2015 at 12:19:01PM -0400, Vadim Bobrov wrote: > > Thanks, Joel, I will but regardless of my findings the basic problem will > > still be there: there is no guarantee that the offsets will be committed > > after commitOffsets. Because commitOffsets does not return its exit > status, > > nor does it block as I understand until offsets are committed. In other > > words, there is no way to know that it has, in fact, commited the offsets > > > > or am I missing something? And then another question - why does it seem > to > > depend on the number of consumed messages? > > > > On Tue, Jul 14, 2015 at 11:36 AM, Joel Koshy <jjkosh...@gmail.com> > wrote: > > > > > Can you take a look at the kafka commit rate mbean on your consumer? > > > Also, can you consume the offsets topic while you are committing > > > offsets and see if/what offsets are getting committed? > > > (http://www.slideshare.net/jjkoshy/offset-management-in-kafka/32) > > > > > > Thanks, > > > > > > Joel > > > > > > On Tue, Jul 14, 2015 at 11:12:03AM -0400, Vadim Bobrov wrote: > > > > I am trying to replace ActiveMQ with Kafka in our environment > however I > > > > have encountered a strange problem that basically prevents from using > > > Kafka > > > > in production. The problem is that sometimes the offsets are not > > > committed. > > > > > > > > I am using Kafka 0.8.2.1, offset storage = kafka, high level > consumer, > > > > auto-commit = off. Every N messages I issue commitOffsets(). Now > here is > > > > the problem - if N is below a certain number (180 000 for me) it > works > > > and > > > > the offset is moving. If N is 180 000 or more the offset is not > updated > > > > after commitOffsets > > > > > > > > I am looking at offsets using kafka-run-class.sh > > > > kafka.tools.ConsumerOffsetChecker > > > > Any help? > > > > > > > >
