Is there anything on the broker log? Is it possible that your client and broker are not running on the same version?
Jiangjie (Becket) Qin On 7/15/15, 11:40 AM, "Vadim Bobrov" <vadimbob...@gmail.com> wrote: >caught it, thanks for help! >any ideas what to do? > >TRACE 2015-07-15 18:37:58,070 [chaos-akka.actor.jms-dispatcher-1019 ] >kafka.network.BoundedByteBufferSend - 113 bytes written. >ERROR 2015-07-15 18:37:58,078 [chaos-akka.actor.jms-dispatcher-1019 ] >kafka.consumer.ZookeeperConsumerConnector - >[chaos-test_ds2.outsmartinc.com-1436984542831-4d399e71], Error while >committing offsets. >java.io.EOFException: Received -1 when reading from channel, socket has >likely been closed. > at kafka.utils.Utils$.read(Utils.scala:381) > at >kafka.network.BoundedByteBufferReceive.readFrom(BoundedByteBufferReceive.s >cala:54) > at >kafka.network.Receive$class.readCompletely(Transmission.scala:56) > at >kafka.network.BoundedByteBufferReceive.readCompletely(BoundedByteBufferRec >eive.scala:29) > at >kafka.network.BlockingChannel.receive(BlockingChannel.scala:111) > at >kafka.consumer.ZookeeperConsumerConnector.liftedTree2$1(ZookeeperConsumerC >onnector.scala:313) > at >kafka.consumer.ZookeeperConsumerConnector.commitOffsets(ZookeeperConsumerC >onnector.scala:310) > at >kafka.javaapi.consumer.ZookeeperConsumerConnector.commitOffsets(ZookeeperC >onsumerConnector.scala:111) > at >com.os.messaging.kafka.KafkaMessageService$class.acknowledge(KafkaMessageS >ervice.scala:55) > at com.os.actor.Main$$anonfun$2$$anon$2.acknowledge(Main.scala:84) > at >com.os.actor.acquisition.MessageListenerActor$$anonfun$8.applyOrElse(Messa >geListenerActor.scala:180) > at >com.os.actor.acquisition.MessageListenerActor$$anonfun$8.applyOrElse(Messa >geListenerActor.scala:164) > at scala.PartialFunction$OrElse.apply(PartialFunction.scala:162) > at akka.actor.FSM$class.processEvent(FSM.scala:607) > at >com.os.actor.acquisition.MessageListenerActor.processEvent(MessageListener >Actor.scala:32) > at akka.actor.FSM$class.akka$actor$FSM$$processMsg(FSM.scala:598) > at akka.actor.FSM$$anonfun$receive$1.applyOrElse(FSM.scala:592) > at akka.actor.Actor$class.aroundReceive(Actor.scala:465) > at >com.os.actor.acquisition.MessageListenerActor.aroundReceive(MessageListene >rActor.scala:32) > at akka.actor.ActorCell.receiveMessage(ActorCell.scala:516) > at akka.actor.ActorCell.invoke(ActorCell.scala:487) > at akka.dispatch.Mailbox.processMailbox(Mailbox.scala:254) > at akka.dispatch.Mailbox.run(Mailbox.scala:221) > at akka.dispatch.Mailbox.exec(Mailbox.scala:231) > at >scala.concurrent.forkjoin.ForkJoinTask.doExec(ForkJoinTask.java:260) > at >scala.concurrent.forkjoin.ForkJoinPool$WorkQueue.runTask(ForkJoinPool.java >:1339) > at >scala.concurrent.forkjoin.ForkJoinPool.runWorker(ForkJoinPool.java:1979) > at >scala.concurrent.forkjoin.ForkJoinWorkerThread.run(ForkJoinWorkerThread.ja >va:107) > > >On Wed, Jul 15, 2015 at 1:36 PM, Vadim Bobrov <vadimbob...@gmail.com> >wrote: > >> thanks Joel and Jiangjie, >> I have figured it out. In addition to my log4j 2 config file I also >>needed >> a log4j 1 config file, then it works. Let me trace what happens when the >> offsets are not committed and report back >> >> On Wed, Jul 15, 2015 at 1:33 PM, Joel Koshy <jjkosh...@gmail.com> wrote: >> >>> - You can also change the log4j level dynamically via the >>> kafka.Log4jController mbean. >>> - You can also look at offset commit request metrics (mbeans) on the >>> broker (just to check if _any_ offset commits are coming through >>> during the period you see no moving offsets). >>> - The alternative is to just consume the offsets topic. >>> >>> On Wed, Jul 15, 2015 at 05:30:17PM +0000, Jiangjie Qin wrote: >>> > I am not sure how your project was setup. But I think it depends on >>>what >>> > log4j property file you specified when you started your application. >>>Can >>> > you check if you have log4j appender defined and the loggers are >>> directed >>> > to the correct appender? >>> > >>> > Thanks, >>> > >>> > Jiangjie (Becket) Qin >>> > >>> > On 7/15/15, 8:10 AM, "Vadim Bobrov" <vadimbob...@gmail.com> wrote: >>> > >>> > >Thanks Jiangjie, >>> > > >>> > >unfortunately turning trace level on does not seem to work (any log >>> level >>> > >actually) I am using log4j2 (through slf4j) and despite including >>> log4j1 >>> > >bridge and these lines: >>> > > >>> > ><Logger name="org.apache.kafka" level="trace"/> >>> > ><Logger name="kafka" level="trace"/> >>> > > >>> > >in my conf file I could not squeeze out any logging from kafka. >>>Logging >>> > >for >>> > >all other libs (like zookeeper e.g.) work perfectly. Am I doing >>> something >>> > >wrong? >>> > > >>> > > >>> > >On Tue, Jul 14, 2015 at 6:55 PM, Jiangjie Qin >>> <j...@linkedin.com.invalid> >>> > >wrote: >>> > > >>> > >> Hi Vadim, >>> > >> >>> > >> Can you turn on trace level logging on your consumer and search >>>for >>> > >> "offset commit response² in the log? >>> > >> Also maybe take a look at the log to see if there is any exception >>> > >>thrown. >>> > >> >>> > >> Thanks, >>> > >> >>> > >> Jiangjie (Becket) Qin >>> > >> >>> > >> On 7/14/15, 11:06 AM, "Vadim Bobrov" <vadimbob...@gmail.com> >>>wrote: >>> > >> >>> > >> >just caught this error again. I issue commitOffsets - no error >>>but >>> no >>> > >> >committng offsets either. __consumer_offsets watching shows no >>>new >>> > >> >messages >>> > >> >either. Then in a few minutes I issue commitOffsets again - all >>> > >>committed. >>> > >> >Unless I am doing something terribly wrong this is very >>>unreliable >>> > >> > >>> > >> >On Tue, Jul 14, 2015 at 1:49 PM, Joel Koshy <jjkosh...@gmail.com> >>> > >>wrote: >>> > >> > >>> > >> >> Actually, how are you committing offsets? Are you using the old >>> > >> >> (zookeeperconsumerconnector) or new KafkaConsumer? >>> > >> >> >>> > >> >> It is true that the current APIs don't return any result, but >>>it >>> > >>would >>> > >> >> help to check if anything is getting into the offsets topic - >>> unless >>> > >> >> you are seeing errors in the logs, the offset commit should >>> succeed >>> > >> >> (if you are indeed explicitly committing offsets). >>> > >> >> >>> > >> >> Thanks, >>> > >> >> >>> > >> >> Joel >>> > >> >> >>> > >> >> On Tue, Jul 14, 2015 at 12:19:01PM -0400, Vadim Bobrov wrote: >>> > >> >> > Thanks, Joel, I will but regardless of my findings the basic >>> > >>problem >>> > >> >>will >>> > >> >> > still be there: there is no guarantee that the offsets will >>>be >>> > >> >>committed >>> > >> >> > after commitOffsets. Because commitOffsets does not return >>>its >>> exit >>> > >> >> status, >>> > >> >> > nor does it block as I understand until offsets are >>>committed. >>> In >>> > >> >>other >>> > >> >> > words, there is no way to know that it has, in fact, commited >>> the >>> > >> >>offsets >>> > >> >> > >>> > >> >> > or am I missing something? And then another question - why >>>does >>> it >>> > >> >>seem >>> > >> >> to >>> > >> >> > depend on the number of consumed messages? >>> > >> >> > >>> > >> >> > On Tue, Jul 14, 2015 at 11:36 AM, Joel Koshy < >>> jjkosh...@gmail.com> >>> > >> >> wrote: >>> > >> >> > >>> > >> >> > > Can you take a look at the kafka commit rate mbean on your >>> > >>consumer? >>> > >> >> > > Also, can you consume the offsets topic while you are >>> committing >>> > >> >> > > offsets and see if/what offsets are getting committed? >>> > >> >> > > ( >>> http://www.slideshare.net/jjkoshy/offset-management-in-kafka/32) >>> > >> >> > > >>> > >> >> > > Thanks, >>> > >> >> > > >>> > >> >> > > Joel >>> > >> >> > > >>> > >> >> > > On Tue, Jul 14, 2015 at 11:12:03AM -0400, Vadim Bobrov >>>wrote: >>> > >> >> > > > I am trying to replace ActiveMQ with Kafka in our >>> environment >>> > >> >> however I >>> > >> >> > > > have encountered a strange problem that basically >>>prevents >>> from >>> > >> >>using >>> > >> >> > > Kafka >>> > >> >> > > > in production. The problem is that sometimes the offsets >>>are >>> > >>not >>> > >> >> > > committed. >>> > >> >> > > > >>> > >> >> > > > I am using Kafka 0.8.2.1, offset storage = kafka, high >>>level >>> > >> >> consumer, >>> > >> >> > > > auto-commit = off. Every N messages I issue >>>commitOffsets(). >>> > >>Now >>> > >> >> here is >>> > >> >> > > > the problem - if N is below a certain number (180 000 for >>> me) >>> > >>it >>> > >> >> works >>> > >> >> > > and >>> > >> >> > > > the offset is moving. If N is 180 000 or more the offset >>>is >>> not >>> > >> >> updated >>> > >> >> > > > after commitOffsets >>> > >> >> > > > >>> > >> >> > > > I am looking at offsets using kafka-run-class.sh >>> > >> >> > > > kafka.tools.ConsumerOffsetChecker >>> > >> >> > > > Any help? >>> > >> >> > > >>> > >> >> > > >>> > >> >> >>> > >> >> >>> > >> >>> > >> >>> > >>> >>> -- >>> Joel >>> >> >>
