I will try to reproduce by repeating the steps I remember, the next time when I restart the cluster.
Thanks Zakee > On Mar 13, 2015, at 4:34 PM, Jiangjie Qin <j...@linkedin.com.INVALID> wrote: > > Can you reproduce this problem? Although the the fix is strait forward we > would like to understand why this happened. > > On 3/13/15, 3:56 PM, "Zakee" <kzak...@netzero.net > <mailto:kzak...@netzero.net>> wrote: > >> Just found there is a known issue to be resolved in future kafka version: >> https://issues.apache.org/jira/browse/KAFKA-1554 >> >> The workaround mentioned here helped. >> >>> The workaround is to delete all index files of size 10MB (the size of >>> the pre-allocated files), and restart. Index files will be re-created. >> >>> find $your_data_directory -size 10485760c -name *.index #-delete >> >> >> Thanks >> Zakee >> >> >> >>> On Mar 13, 2015, at 3:38 PM, Zakee <kzak...@netzero.net> wrote: >>> >>> I did a shutdown of the cluster and then try to restart and see the >>> below error on one of the 5 brokers, I can¹t restart this instance and >>> not sure how to fix this. >>> >>> [2015-03-13 15:27:31,793] ERROR There was an error in one of the >>> threads during logs loading: java.lang.IllegalArgumentException: >>> requirement failed: Corrupt index found, index file >>> (/data/vol6/kafka/kafka82/Topic22-6/00000000000006256447.index) has >>> non-zero size but the last offset is 6256447 and the base offset is >>> 6256447 (kafka.log.LogManager) >>> [2015-03-13 15:27:31,795] FATAL [Kafka Server 5], Fatal error during >>> KafkaServer startup. Prepare to shutdown (kafka.server.KafkaServer) >>> java.lang.IllegalArgumentException: requirement failed: Corrupt index >>> found, index file >>> (/data/vol6/kafka/kafka82/Topic22-6/00000000000006256447.index) has >>> non-zero size but the last offset is 6256447 and the base offset is >>> 6256447 >>> at scala.Predef$.require(Predef.scala:233) >>> at kafka.log.OffsetIndex.sanityCheck(OffsetIndex.scala:352) >>> at kafka.log.Log$$anonfun$loadSegments$5.apply(Log.scala:184) >>> at kafka.log.Log$$anonfun$loadSegments$5.apply(Log.scala:183) >>> at scala.collection.Iterator$class.foreach(Iterator.scala:727) >>> at scala.collection.AbstractIterator.foreach(Iterator.scala:1157) >>> at >>> scala.collection.IterableLike$class.foreach(IterableLike.scala:72) >>> at scala.collection.AbstractIterable.foreach(Iterable.scala:54) >>> at kafka.log.Log.loadSegments(Log.scala:183) >>> at kafka.log.Log.<init>(Log.scala:67) >>> at >>> kafka.log.LogManager$$anonfun$loadLogs$2$$anonfun$3$$anonfun$apply$7$$ano >>> nfun$apply$1.apply$mcV$sp(LogManager.scala:142) >>> at kafka.utils.Utils$$anon$1.run(Utils.scala:54) >>> at >>> java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:441) >>> at >>> java.util.concurrent.FutureTask$Sync.innerRun(FutureTask.java:303) >>> at java.util.concurrent.FutureTask.run(FutureTask.java:138) >>> at >>> java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor >>> .java:886) >>> at >>> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.jav >>> a:908) >>> at java.lang.Thread.run(Thread.java:662) >>> [2015-03-13 15:27:31,831] INFO [Kafka Server 5], shutting down >>> (kafka.server.KafkaServer) >>> >>> >>> Thanks >>> Zakee >>> >>> >>> >> >> ____________________________________________________________ >> Protect what matters >> Floods can happen anywhere. Learn your risk and find an agent today. >> http://thirdpartyoffers.netzero.net/TGL3231/55036b26df3ee6b257d4fst02duc >> <http://thirdpartyoffers.netzero.net/TGL3231/55036b26df3ee6b257d4fst02duc> > > ____________________________________________________________ > The WORST exercise for aging > Avoid this "healthy" exercise to look & feel 5-10 years YOUNGER > http://thirdpartyoffers.netzero.net/TGL3255/5503813e152c713e64c2mp13duc > <http://thirdpartyoffers.netzero.net/TGL3255/5503813e152c713e64c2mp13duc>
