I am testing compact policy on a topic and got the following exception in log-cleaner.log. It seems to be related to the size of the ByteBuffer. Has anyone seen this error before, or any config I can tune to increase this?
[2016-01-21 04:21:23,083] INFO Cleaner 0: Beginning cleaning of log topic-0. (kafka.log.LogCleaner) [2016-01-21 04:21:23,083] INFO Cleaner 0: Building offset map for topic-0... (kafka.log.LogCleaner) [2016-01-21 04:21:23,148] INFO Cleaner 0: Building offset map for log topic-0 for 6 segments in offset range [0, 56070). (kafka.log.LogCleaner) [2016-01-21 04:21:23,441] INFO Cleaner 0: Offset map for log topic-0 complete. (kafka.log.LogCleaner) [2016-01-21 04:21:23,442] INFO Cleaner 0: Cleaning log topic-0 (discarding tombstones prior to Thu Jan 01 00:00:00 UTC 1970)... (kafka.log.LogCleaner) [2016-01-21 04:21:23,442] INFO Cleaner 0: Cleaning segment 0 in log topic-0 (last modified Wed Jan 20 18:38:33 UTC 2016) into 0, retaining deletes. (kafka.log.LogCleaner) [2016-01-21 04:21:23,455] INFO Cleaner 0: Cleaning segment 6230 in log topic-0 (last modified Wed Jan 20 20:27:59 UTC 2016) into 0, retaining deletes. (kafka.log.LogCleaner) [2016-01-21 04:21:23,467] INFO Cleaner 0: Cleaning segment 18690 in log topic-0 (last modified Wed Jan 20 21:26:58 UTC 2016) into 0, retaining deletes. (kafka.log.LogCleaner) [2016-01-21 04:21:23,479] INFO Cleaner 0: Cleaning segment 24920 in log topic-0 (last modified Wed Jan 20 23:26:48 UTC 2016) into 0, retaining deletes. (kafka.log.LogCleaner) [2016-01-21 04:21:23,489] INFO Cleaner 0: Cleaning segment 37380 in log topic-0 (last modified Thu Jan 21 00:25:16 UTC 2016) into 0, retaining deletes. (kafka.log.LogCleaner) [2016-01-21 04:21:23,495] INFO Cleaner 0: Cleaning segment 43610 in log topic-0 (last modified Thu Jan 21 02:26:17 UTC 2016) into 0, retaining deletes. (kafka.log.LogCleaner) [2016-01-21 04:21:23,513] ERROR [kafka-log-cleaner-thread-0], Error due to (kafka.log.LogCleaner) java.nio.BufferOverflowException at java.nio.Buffer.nextPutIndex(Buffer.java:519) at java.nio.HeapByteBuffer.putLong(HeapByteBuffer.java:417) at kafka.message.ByteBufferMessageSet$.writeMessage(ByteBufferMessageSet.scala:80) at kafka.log.Cleaner$$anonfun$cleanInto$1.apply(LogCleaner.scala:419) at kafka.log.Cleaner$$anonfun$cleanInto$1.apply(LogCleaner.scala:404) at scala.collection.Iterator$class.foreach(Iterator.scala:727) at kafka.utils.IteratorTemplate.foreach(IteratorTemplate.scala:32) at scala.collection.IterableLike$class.foreach(IterableLike.scala:72) at kafka.message.MessageSet.foreach(MessageSet.scala:67) at kafka.log.Cleaner.cleanInto(LogCleaner.scala:404) at kafka.log.Cleaner$$anonfun$cleanSegments$1.apply(LogCleaner.scala:358) at kafka.log.Cleaner$$anonfun$cleanSegments$1.apply(LogCleaner.scala:354) at scala.collection.immutable.List.foreach(List.scala:318) at kafka.log.Cleaner.cleanSegments(LogCleaner.scala:354) at kafka.log.Cleaner$$anonfun$clean$4.apply(LogCleaner.scala:321) at kafka.log.Cleaner$$anonfun$clean$4.apply(LogCleaner.scala:320) at scala.collection.immutable.List.foreach(List.scala:318) -- Chen Song