Hello, We are using Kafka 0.8.2.2 and have modified most of our topics to use log compaction and a shorter retention.ms equivalent to 24 hours for those topics. We did not see any change in the disk usage, and on looking at the log cleaner log file, noticed the following message (repeated on each restart of kafka)
[2016-03-07 15:35:20,931] INFO Starting the log cleaner (kafka.log.LogCleaner) [2016-03-07 15:35:20,933] INFO [kafka-log-cleaner-thread-0], Starting (kafka.log.LogCleaner) [2016-03-07 15:35:20,954] INFO Cleaner 0: Beginning cleaning of log metamorphosis.lineup-0. (kafka.log.LogCleaner) [2016-03-07 15:35:20,954] INFO Cleaner 0: Building offset map for metamorphosis.lineup-0... (kafka.log.LogCleaner) [2016-03-07 15:35:21,047] INFO Cleaner 0: Building offset map for log metamorphosis.lineup-0 for 14 segments in offset range [1553258, 1939758). (kafka.log.LogCleaner) [2016-03-07 15:35:21,080] ERROR [kafka-log-cleaner-thread-0], Error due to (kafka.log.LogCleaner) java.lang.IllegalArgumentException: requirement failed at scala.Predef$.require(Predef.scala:221) at kafka.log.Cleaner$$anonfun$kafka$log$Cleaner$$buildOffsetMapForSegment$1.apply(LogCleaner.scala:539) at kafka.log.Cleaner$$anonfun$kafka$log$Cleaner$$buildOffsetMapForSegment$1.apply(LogCleaner.scala:537) at scala.collection.Iterator$class.foreach(Iterator.scala:727) at kafka.utils.IteratorTemplate.foreach(IteratorTemplate.scala:32) at scala.collection.IterableLike$class.foreach(IterableLike.scala:72) at kafka.message.MessageSet.foreach(MessageSet.scala:67) at kafka.log.Cleaner.kafka$log$Cleaner$$buildOffsetMapForSegment(LogCleaner.scala:537) at kafka.log.Cleaner$$anonfun$buildOffsetMap$3.apply(LogCleaner.scala:514) at kafka.log.Cleaner$$anonfun$buildOffsetMap$3.apply(LogCleaner.scala:511) at scala.collection.immutable.Stream.foreach(Stream.scala:547) at kafka.log.Cleaner.buildOffsetMap(LogCleaner.scala:511) at kafka.log.Cleaner.clean(LogCleaner.scala:307) at kafka.log.LogCleaner$CleanerThread.cleanOrSleep(LogCleaner.scala:221) at kafka.log.LogCleaner$CleanerThread.doWork(LogCleaner.scala:199) at kafka.utils.ShutdownableThread.run(ShutdownableThread.scala:60) [2016-03-07 15:35:21,083] INFO [kafka-log-cleaner-thread-0], Stopped (kafka.log.LogCleaner) I tried setting a higher value for the log cleaner dedupe buffer size to see if that was the issue, but it did not make any difference (log.cleaner.dedupe.buffer.size=536870912) Any ideas as to the cause of this error? Is it due to some wrong configuration? Thanks Rakesh
