[
https://issues.apache.org/jira/browse/KAFKA-1791?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14221851#comment-14221851
]
Vamsi Subhash Achanta commented on KAFKA-1791:
----------------------------------------------
Attached the file.
> Corrupt index after safe shutdown and restart
> ---------------------------------------------
>
> Key: KAFKA-1791
> URL: https://issues.apache.org/jira/browse/KAFKA-1791
> Project: Kafka
> Issue Type: Bug
> Components: core
> Affects Versions: 0.8.1
> Environment: Debian6 with Sun-Java6
> Reporter: Vamsi Subhash Achanta
> Priority: Critical
> Attachments: 00000000000000000233.index
>
>
> We have 3 kafka brokers - all VMs. One of the broker was stopped for around
> 30 minutes to fix a problem with the bare metal. Upon restart, after some
> time, the broker went out of file-descriptors (FDs) and started throwing
> errors. Upon restart, it started throwing this corrupted index exceptions. I
> followed the other JIRAs related to corrupted indices but the solutions
> mentioned there (deleting the index and restart) didn't work - the index gets
> created again. The other JIRAs solution of deleting those indexes which got
> wrongly compacted (> 10MB) didn't work either.
> What is the error? How can I fix this and bring back the broker? Thanks.
> INFO [2014-11-21 02:57:17,510] [main][] kafka.log.LogManager - Found clean
> shutdown file. Skipping recovery for all logs in data directory
> '/var/lib/fk-3p-kafka/logs'
> INFO [2014-11-21 02:57:17,510] [main][] kafka.log.LogManager - Loading log
> 'kf.production.b2b.return_order.status-25'
> FATAL [2014-11-21 02:57:17,533] [main][] kafka.server.KafkaServerStartable -
> Fatal error during KafkaServerStable startup. Prepare to shutdown
> java.lang.IllegalArgumentException: requirement failed: Corrupt index found,
> index file
> (/var/lib/fk-3p-kafka/logs/kf.production.b2b.return_order.status-25/00000000000000000233.index)
> has non-zero size but the last offset is 233 and the base offset is 233
> at scala.Predef$.require(Predef.scala:145)
> at kafka.log.OffsetIndex.sanityCheck(OffsetIndex.scala:352)
> at kafka.log.Log$$anonfun$loadSegments$5.apply(Log.scala:159)
> at kafka.log.Log$$anonfun$loadSegments$5.apply(Log.scala:158)
> at scala.collection.Iterator$class.foreach(Iterator.scala:631)
> at
> scala.collection.JavaConversions$JIteratorWrapper.foreach(JavaConversions.scala:474)
> at scala.collection.IterableLike$class.foreach(IterableLike.scala:79)
> at
> scala.collection.JavaConversions$JCollectionWrapper.foreach(JavaConversions.scala:495)
> at kafka.log.Log.loadSegments(Log.scala:158)
> at kafka.log.Log.<init>(Log.scala:64)
> at
> kafka.log.LogManager$$anonfun$loadLogs$1$$anonfun$apply$4.apply(LogManager.scala:118)
> at
> kafka.log.LogManager$$anonfun$loadLogs$1$$anonfun$apply$4.apply(LogManager.scala:113)
> at
> scala.collection.IndexedSeqOptimized$class.foreach(IndexedSeqOptimized.scala:34)
> at scala.collection.mutable.ArrayOps.foreach(ArrayOps.scala:34)
> at kafka.log.LogManager$$anonfun$loadLogs$1.apply(LogManager.scala:113)
> at kafka.log.LogManager$$anonfun$loadLogs$1.apply(LogManager.scala:105)
> at
> scala.collection.IndexedSeqOptimized$class.foreach(IndexedSeqOptimized.scala:34)
> at scala.collection.mutable.WrappedArray.foreach(WrappedArray.scala:32)
> at kafka.log.LogManager.loadLogs(LogManager.scala:105)
> at kafka.log.LogManager.<init>(LogManager.scala:57)
> at kafka.server.KafkaServer.createLogManager(KafkaServer.scala:275)
> at kafka.server.KafkaServer.startup(KafkaServer.scala:72)
> at
> kafka.server.KafkaServerStartable.startup(KafkaServerStartable.scala:34)
> at kafka.Kafka$.main(Kafka.scala:46)
> at kafka.Kafka.main(Kafka.scala)
> INFO [2014-11-21 02:57:17,534] [main][] kafka.server.KafkaServer - [Kafka
> Server 2], shutting down
> INFO [2014-11-21 02:57:17,538] [main][] kafka.server.KafkaServer - [Kafka
> Server 2], shut down completed
> INFO [2014-11-21 02:57:17,539] [Thread-2][] kafka.server.KafkaServer -
> [Kafka Server 2], shutting down
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
