We decided to clear the old logs and start fresh. I'll run that app the next time if we run into the same problem again.
Thanks Jun, Xuyen -----Original Message----- From: Jun Rao [mailto:jun...@gmail.com] Sent: Monday, May 12, 2014 8:58 PM To: users@kafka.apache.org Subject: Re: IndexOutOfBoundsException error Could you run the DumpLogSegments tool to see if there are indeed corrupted messages in the broker log? Thanks, Jun On Mon, May 12, 2014 at 4:08 PM, Xuyen On <x...@ancestry.com> wrote: > Hi all, > > I am seeing some corrupt data on my 0.7.2 Kafka cluster. Every once in > awhile I'll get the following message: > > Exception in thread "kafka-consumer" java.lang.IndexOutOfBoundsException > at java.nio.Buffer.checkIndex(Unknown Source) > at java.nio.HeapByteBuffer.get(Unknown Source) > at kafka.message.Message.magic(Message.scala:133) > at kafka.message.Message.checksum(Message.scala:146) > at kafka.message.Message.isValid(Message.scala:158) > at > kafka.message.ByteBufferMessageSet$$anon$1.makeNextOuter(ByteBufferMessageSet.scala:129) > at > kafka.message.ByteBufferMessageSet$$anon$1.makeNext(ByteBufferMessageSet.scala:166) > at > kafka.message.ByteBufferMessageSet$$anon$1.makeNext(ByteBufferMessageSet.scala:100) > at > kafka.utils.IteratorTemplate.maybeComputeNext(IteratorTemplate.scala:59) > at kafka.utils.IteratorTemplate.hasNext(IteratorTemplate.scala:51) > at scala.collection.Iterator$class.foreach(Iterator.scala:631) > at kafka.utils.IteratorTemplate.foreach(IteratorTemplate.scala:30) > at > scala.collection.IterableLike$class.foreach(IterableLike.scala:79) > at kafka.message.MessageSet.foreach(MessageSet.scala:87) > at > kafka.tools.SimpleConsumerShell$$anon$1$$anonfun$run$1.apply(SimpleConsumerShell.scala:97) > at > kafka.tools.SimpleConsumerShell$$anon$1$$anonfun$run$1.apply(SimpleConsumerShell.scala:94) > at scala.collection.Iterator$class.foreach(Iterator.scala:631) > at kafka.utils.IteratorTemplate.foreach(IteratorTemplate.scala:30) > at > scala.collection.IterableLike$class.foreach(IterableLike.scala:79) > at > kafka.api.MultiFetchResponse.foreach(MultiFetchResponse.scala:25) > at > kafka.tools.SimpleConsumerShell$$anon$1.run(SimpleConsumerShell.scala: > 94) > > The only way I can get past this error is to move the offset past the > current message to skip the bad data. I use the > simpleConsumer.getOffsetsBefore() method to get a valid lists of > offsets and I skip to the next valid offset. > > Has anyone encountered this issue before? I found a message where > someone experienced similar issues and they were running JDK 1.7 and > was able to get around it by running JDK 1.6. I am running OpenJDK 1.7 > but I tried running Sun JDK 1.6 but I still have the same problem. > > My guess is that the data is corrupt somehow but I don't know how to > look at the raw data to confirm. Can anyone suggest ideas for me to > debug this issue? > > Thanks, > > Xuyen > > >
