Hi Vadim, Can you turn on trace level logging on your consumer and search for "offset commit response² in the log? Also maybe take a look at the log to see if there is any exception thrown.
Thanks, Jiangjie (Becket) Qin On 7/14/15, 11:06 AM, "Vadim Bobrov" <vadimbob...@gmail.com> wrote: >just caught this error again. I issue commitOffsets - no error but no >committng offsets either. __consumer_offsets watching shows no new >messages >either. Then in a few minutes I issue commitOffsets again - all committed. >Unless I am doing something terribly wrong this is very unreliable > >On Tue, Jul 14, 2015 at 1:49 PM, Joel Koshy <jjkosh...@gmail.com> wrote: > >> Actually, how are you committing offsets? Are you using the old >> (zookeeperconsumerconnector) or new KafkaConsumer? >> >> It is true that the current APIs don't return any result, but it would >> help to check if anything is getting into the offsets topic - unless >> you are seeing errors in the logs, the offset commit should succeed >> (if you are indeed explicitly committing offsets). >> >> Thanks, >> >> Joel >> >> On Tue, Jul 14, 2015 at 12:19:01PM -0400, Vadim Bobrov wrote: >> > Thanks, Joel, I will but regardless of my findings the basic problem >>will >> > still be there: there is no guarantee that the offsets will be >>committed >> > after commitOffsets. Because commitOffsets does not return its exit >> status, >> > nor does it block as I understand until offsets are committed. In >>other >> > words, there is no way to know that it has, in fact, commited the >>offsets >> > >> > or am I missing something? And then another question - why does it >>seem >> to >> > depend on the number of consumed messages? >> > >> > On Tue, Jul 14, 2015 at 11:36 AM, Joel Koshy <jjkosh...@gmail.com> >> wrote: >> > >> > > Can you take a look at the kafka commit rate mbean on your consumer? >> > > Also, can you consume the offsets topic while you are committing >> > > offsets and see if/what offsets are getting committed? >> > > (http://www.slideshare.net/jjkoshy/offset-management-in-kafka/32) >> > > >> > > Thanks, >> > > >> > > Joel >> > > >> > > On Tue, Jul 14, 2015 at 11:12:03AM -0400, Vadim Bobrov wrote: >> > > > I am trying to replace ActiveMQ with Kafka in our environment >> however I >> > > > have encountered a strange problem that basically prevents from >>using >> > > Kafka >> > > > in production. The problem is that sometimes the offsets are not >> > > committed. >> > > > >> > > > I am using Kafka 0.8.2.1, offset storage = kafka, high level >> consumer, >> > > > auto-commit = off. Every N messages I issue commitOffsets(). Now >> here is >> > > > the problem - if N is below a certain number (180 000 for me) it >> works >> > > and >> > > > the offset is moving. If N is 180 000 or more the offset is not >> updated >> > > > after commitOffsets >> > > > >> > > > I am looking at offsets using kafka-run-class.sh >> > > > kafka.tools.ConsumerOffsetChecker >> > > > Any help? >> > > >> > > >> >>
