caught it, thanks for help! any ideas what to do? TRACE 2015-07-15 18:37:58,070 [chaos-akka.actor.jms-dispatcher-1019 ] kafka.network.BoundedByteBufferSend - 113 bytes written. ERROR 2015-07-15 18:37:58,078 [chaos-akka.actor.jms-dispatcher-1019 ] kafka.consumer.ZookeeperConsumerConnector - [chaos-test_ds2.outsmartinc.com-1436984542831-4d399e71], Error while committing offsets. java.io.EOFException: Received -1 when reading from channel, socket has likely been closed. at kafka.utils.Utils$.read(Utils.scala:381) at kafka.network.BoundedByteBufferReceive.readFrom(BoundedByteBufferReceive.scala:54) at kafka.network.Receive$class.readCompletely(Transmission.scala:56) at kafka.network.BoundedByteBufferReceive.readCompletely(BoundedByteBufferReceive.scala:29) at kafka.network.BlockingChannel.receive(BlockingChannel.scala:111) at kafka.consumer.ZookeeperConsumerConnector.liftedTree2$1(ZookeeperConsumerConnector.scala:313) at kafka.consumer.ZookeeperConsumerConnector.commitOffsets(ZookeeperConsumerConnector.scala:310) at kafka.javaapi.consumer.ZookeeperConsumerConnector.commitOffsets(ZookeeperConsumerConnector.scala:111) at com.os.messaging.kafka.KafkaMessageService$class.acknowledge(KafkaMessageService.scala:55) at com.os.actor.Main$$anonfun$2$$anon$2.acknowledge(Main.scala:84) at com.os.actor.acquisition.MessageListenerActor$$anonfun$8.applyOrElse(MessageListenerActor.scala:180) at com.os.actor.acquisition.MessageListenerActor$$anonfun$8.applyOrElse(MessageListenerActor.scala:164) at scala.PartialFunction$OrElse.apply(PartialFunction.scala:162) at akka.actor.FSM$class.processEvent(FSM.scala:607) at com.os.actor.acquisition.MessageListenerActor.processEvent(MessageListenerActor.scala:32) at akka.actor.FSM$class.akka$actor$FSM$$processMsg(FSM.scala:598) at akka.actor.FSM$$anonfun$receive$1.applyOrElse(FSM.scala:592) at akka.actor.Actor$class.aroundReceive(Actor.scala:465) at com.os.actor.acquisition.MessageListenerActor.aroundReceive(MessageListenerActor.scala:32) at akka.actor.ActorCell.receiveMessage(ActorCell.scala:516) at akka.actor.ActorCell.invoke(ActorCell.scala:487) at akka.dispatch.Mailbox.processMailbox(Mailbox.scala:254) at akka.dispatch.Mailbox.run(Mailbox.scala:221) at akka.dispatch.Mailbox.exec(Mailbox.scala:231) at scala.concurrent.forkjoin.ForkJoinTask.doExec(ForkJoinTask.java:260) at scala.concurrent.forkjoin.ForkJoinPool$WorkQueue.runTask(ForkJoinPool.java:1339) at scala.concurrent.forkjoin.ForkJoinPool.runWorker(ForkJoinPool.java:1979) at scala.concurrent.forkjoin.ForkJoinWorkerThread.run(ForkJoinWorkerThread.java:107)
On Wed, Jul 15, 2015 at 1:36 PM, Vadim Bobrov <vadimbob...@gmail.com> wrote: > thanks Joel and Jiangjie, > I have figured it out. In addition to my log4j 2 config file I also needed > a log4j 1 config file, then it works. Let me trace what happens when the > offsets are not committed and report back > > On Wed, Jul 15, 2015 at 1:33 PM, Joel Koshy <jjkosh...@gmail.com> wrote: > >> - You can also change the log4j level dynamically via the >> kafka.Log4jController mbean. >> - You can also look at offset commit request metrics (mbeans) on the >> broker (just to check if _any_ offset commits are coming through >> during the period you see no moving offsets). >> - The alternative is to just consume the offsets topic. >> >> On Wed, Jul 15, 2015 at 05:30:17PM +0000, Jiangjie Qin wrote: >> > I am not sure how your project was setup. But I think it depends on what >> > log4j property file you specified when you started your application. Can >> > you check if you have log4j appender defined and the loggers are >> directed >> > to the correct appender? >> > >> > Thanks, >> > >> > Jiangjie (Becket) Qin >> > >> > On 7/15/15, 8:10 AM, "Vadim Bobrov" <vadimbob...@gmail.com> wrote: >> > >> > >Thanks Jiangjie, >> > > >> > >unfortunately turning trace level on does not seem to work (any log >> level >> > >actually) I am using log4j2 (through slf4j) and despite including >> log4j1 >> > >bridge and these lines: >> > > >> > ><Logger name="org.apache.kafka" level="trace"/> >> > ><Logger name="kafka" level="trace"/> >> > > >> > >in my conf file I could not squeeze out any logging from kafka. Logging >> > >for >> > >all other libs (like zookeeper e.g.) work perfectly. Am I doing >> something >> > >wrong? >> > > >> > > >> > >On Tue, Jul 14, 2015 at 6:55 PM, Jiangjie Qin >> <j...@linkedin.com.invalid> >> > >wrote: >> > > >> > >> Hi Vadim, >> > >> >> > >> Can you turn on trace level logging on your consumer and search for >> > >> "offset commit response² in the log? >> > >> Also maybe take a look at the log to see if there is any exception >> > >>thrown. >> > >> >> > >> Thanks, >> > >> >> > >> Jiangjie (Becket) Qin >> > >> >> > >> On 7/14/15, 11:06 AM, "Vadim Bobrov" <vadimbob...@gmail.com> wrote: >> > >> >> > >> >just caught this error again. I issue commitOffsets - no error but >> no >> > >> >committng offsets either. __consumer_offsets watching shows no new >> > >> >messages >> > >> >either. Then in a few minutes I issue commitOffsets again - all >> > >>committed. >> > >> >Unless I am doing something terribly wrong this is very unreliable >> > >> > >> > >> >On Tue, Jul 14, 2015 at 1:49 PM, Joel Koshy <jjkosh...@gmail.com> >> > >>wrote: >> > >> > >> > >> >> Actually, how are you committing offsets? Are you using the old >> > >> >> (zookeeperconsumerconnector) or new KafkaConsumer? >> > >> >> >> > >> >> It is true that the current APIs don't return any result, but it >> > >>would >> > >> >> help to check if anything is getting into the offsets topic - >> unless >> > >> >> you are seeing errors in the logs, the offset commit should >> succeed >> > >> >> (if you are indeed explicitly committing offsets). >> > >> >> >> > >> >> Thanks, >> > >> >> >> > >> >> Joel >> > >> >> >> > >> >> On Tue, Jul 14, 2015 at 12:19:01PM -0400, Vadim Bobrov wrote: >> > >> >> > Thanks, Joel, I will but regardless of my findings the basic >> > >>problem >> > >> >>will >> > >> >> > still be there: there is no guarantee that the offsets will be >> > >> >>committed >> > >> >> > after commitOffsets. Because commitOffsets does not return its >> exit >> > >> >> status, >> > >> >> > nor does it block as I understand until offsets are committed. >> In >> > >> >>other >> > >> >> > words, there is no way to know that it has, in fact, commited >> the >> > >> >>offsets >> > >> >> > >> > >> >> > or am I missing something? And then another question - why does >> it >> > >> >>seem >> > >> >> to >> > >> >> > depend on the number of consumed messages? >> > >> >> > >> > >> >> > On Tue, Jul 14, 2015 at 11:36 AM, Joel Koshy < >> jjkosh...@gmail.com> >> > >> >> wrote: >> > >> >> > >> > >> >> > > Can you take a look at the kafka commit rate mbean on your >> > >>consumer? >> > >> >> > > Also, can you consume the offsets topic while you are >> committing >> > >> >> > > offsets and see if/what offsets are getting committed? >> > >> >> > > ( >> http://www.slideshare.net/jjkoshy/offset-management-in-kafka/32) >> > >> >> > > >> > >> >> > > Thanks, >> > >> >> > > >> > >> >> > > Joel >> > >> >> > > >> > >> >> > > On Tue, Jul 14, 2015 at 11:12:03AM -0400, Vadim Bobrov wrote: >> > >> >> > > > I am trying to replace ActiveMQ with Kafka in our >> environment >> > >> >> however I >> > >> >> > > > have encountered a strange problem that basically prevents >> from >> > >> >>using >> > >> >> > > Kafka >> > >> >> > > > in production. The problem is that sometimes the offsets are >> > >>not >> > >> >> > > committed. >> > >> >> > > > >> > >> >> > > > I am using Kafka 0.8.2.1, offset storage = kafka, high level >> > >> >> consumer, >> > >> >> > > > auto-commit = off. Every N messages I issue commitOffsets(). >> > >>Now >> > >> >> here is >> > >> >> > > > the problem - if N is below a certain number (180 000 for >> me) >> > >>it >> > >> >> works >> > >> >> > > and >> > >> >> > > > the offset is moving. If N is 180 000 or more the offset is >> not >> > >> >> updated >> > >> >> > > > after commitOffsets >> > >> >> > > > >> > >> >> > > > I am looking at offsets using kafka-run-class.sh >> > >> >> > > > kafka.tools.ConsumerOffsetChecker >> > >> >> > > > Any help? >> > >> >> > > >> > >> >> > > >> > >> >> >> > >> >> >> > >> >> > >> >> > >> >> -- >> Joel >> > >