Found the issue. My publisher was not assigning a key to all messages.
On 07/03/2016 14:40, "Rakesh Vidyadharan" <rvidyadha...@gracenote.com> wrote: >Hello, > >We are using Kafka 0.8.2.2 and have modified most of our topics to use log >compaction and a shorter retention.ms equivalent to 24 hours for those topics. > We did not see any change in the disk usage, and on looking at the log >cleaner log file, noticed the following message (repeated on each restart of >kafka) > >[2016-03-07 15:35:20,931] INFO Starting the log cleaner (kafka.log.LogCleaner) >[2016-03-07 15:35:20,933] INFO [kafka-log-cleaner-thread-0], Starting >(kafka.log.LogCleaner) >[2016-03-07 15:35:20,954] INFO Cleaner 0: Beginning cleaning of log >metamorphosis.lineup-0. (kafka.log.LogCleaner) >[2016-03-07 15:35:20,954] INFO Cleaner 0: Building offset map for >metamorphosis.lineup-0... (kafka.log.LogCleaner) >[2016-03-07 15:35:21,047] INFO Cleaner 0: Building offset map for log >metamorphosis.lineup-0 for 14 segments in offset range [1553258, 1939758). >(kafka.log.LogCleaner) >[2016-03-07 15:35:21,080] ERROR [kafka-log-cleaner-thread-0], Error due to >(kafka.log.LogCleaner) >java.lang.IllegalArgumentException: requirement failed >at scala.Predef$.require(Predef.scala:221) >at >kafka.log.Cleaner$$anonfun$kafka$log$Cleaner$$buildOffsetMapForSegment$1.apply(LogCleaner.scala:539) >at >kafka.log.Cleaner$$anonfun$kafka$log$Cleaner$$buildOffsetMapForSegment$1.apply(LogCleaner.scala:537) >at scala.collection.Iterator$class.foreach(Iterator.scala:727) >at kafka.utils.IteratorTemplate.foreach(IteratorTemplate.scala:32) >at scala.collection.IterableLike$class.foreach(IterableLike.scala:72) >at kafka.message.MessageSet.foreach(MessageSet.scala:67) >at >kafka.log.Cleaner.kafka$log$Cleaner$$buildOffsetMapForSegment(LogCleaner.scala:537) >at kafka.log.Cleaner$$anonfun$buildOffsetMap$3.apply(LogCleaner.scala:514) >at kafka.log.Cleaner$$anonfun$buildOffsetMap$3.apply(LogCleaner.scala:511) >at scala.collection.immutable.Stream.foreach(Stream.scala:547) >at kafka.log.Cleaner.buildOffsetMap(LogCleaner.scala:511) >at kafka.log.Cleaner.clean(LogCleaner.scala:307) >at kafka.log.LogCleaner$CleanerThread.cleanOrSleep(LogCleaner.scala:221) >at kafka.log.LogCleaner$CleanerThread.doWork(LogCleaner.scala:199) >at kafka.utils.ShutdownableThread.run(ShutdownableThread.scala:60) >[2016-03-07 15:35:21,083] INFO [kafka-log-cleaner-thread-0], Stopped >(kafka.log.LogCleaner) > >I tried setting a higher value for the log cleaner dedupe buffer size to see >if that was the issue, but it did not make any difference >(log.cleaner.dedupe.buffer.size=536870912) > >Any ideas as to the cause of this error? Is it due to some wrong >configuration? > >Thanks >Rakesh
