[
https://issues.apache.org/jira/browse/KAFKA-7951?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16774132#comment-16774132
]
Ninth Nails commented on KAFKA-7951:
------------------------------------
Hi [~huxi_2b],
I didn't upgraded or downgraded Kafka before having this problem. I did tried
to upgrade _after_ though, to version 0.11.0.3, but it didn't fix the issue.
Interestingly, no other broker log cleaner have failed so far, even though the
partition has been moved.
Regarding the record size, what is the expect maximum value?
Thanks
> Log Cleaner thread stop with "Varint is too long" error
> -------------------------------------------------------
>
> Key: KAFKA-7951
> URL: https://issues.apache.org/jira/browse/KAFKA-7951
> Project: Kafka
> Issue Type: Bug
> Components: log
> Affects Versions: 0.11.0.3
> Environment: Amazon Linux AMI 2017.09
> confluent-platform-oss-2.11-3.3.2
> OpenJDK Runtime Environment (build 1.8.0_191-b12)
> EC2 instance type i3.xlarge
> Reporter: Ninth Nails
> Priority: Blocker
>
> I have one broker log cleaner thread dying because of a failing type
> conversion. My cluster is running on 0.11.03-cp1 (Confluent Platform). Here's
> the relevant logs in log-cleaner.log
> {code:java}
> [2019-02-19 15:08:23,387] priority=INFO message="[kafka-log-cleaner-thread-0]:
> Log cleaner thread 0 cleaned log __consumer_offsets-36 (dirty section
> = [931148954, 931148954])
> 13,051.5 MB of log processed in 159.0 seconds (82.1 MB/sec).
> Indexed 13,051.5 MB in 81.2 seconds (160.8 Mb/sec, 51.1% of total
> time)
> Buffer utilization: 0.0%
> Cleaned 13,051.5 MB in 77.8 seconds (167.7 Mb/sec, 48.9% of total
> time)
> Start size: 13,051.5 MB (166,676,749 messages)
> End size: 0.0 MB (159 messages)
> 100.0% size reduction (100.0% fewer messages)
> " category=kafka.log.LogCleaner
> [2019-02-19 15:08:23,392] priority=INFO message="Cleaner 0: Beginning
> cleaning of log __consumer_offsets-37." category=kafka.log.LogCleaner
> [2019-02-19 15:08:23,392] priority=INFO message="Cleaner 0: Building offset
> map for __consumer_offsets-37..." category=kafka.log.LogCleaner
> [2019-02-19 15:08:23,410] priority=INFO message="Cleaner 0: Building offset
> map for log __consumer_offsets-37 for 87 segments in offset range [211566448,
> 308245257)." category=kafka.log.LogCleaner
> [2019-02-19 15:08:23,945] priority=ERROR
> message="[kafka-log-cleaner-thread-0]: Error due to"
> category=kafka.log.LogCleaner
> java.lang.IllegalArgumentException: Varint is too long, the most significant
> bit in the 5th byte is set, converted value: 445d418c
> at
> org.apache.kafka.common.utils.ByteUtils.illegalVarintException(ByteUtils.java:326)
> at
> org.apache.kafka.common.utils.ByteUtils.readVarint(ByteUtils.java:148)
> at
> org.apache.kafka.common.record.DefaultRecord.readFrom(DefaultRecord.java:305)
> at
> org.apache.kafka.common.record.DefaultRecordBatch$2.readNext(DefaultRecordBatch.java:299)
> at
> org.apache.kafka.common.record.DefaultRecordBatch$RecordIterator.next(DefaultRecordBatch.java:563)
> at
> org.apache.kafka.common.record.DefaultRecordBatch$RecordIterator.next(DefaultRecordBatch.java:532)
> at
> scala.collection.convert.Wrappers$JIteratorWrapper.next(Wrappers.scala:43)
> at scala.collection.Iterator$class.foreach(Iterator.scala:891)
> at scala.collection.AbstractIterator.foreach(Iterator.scala:1334)
> at scala.collection.IterableLike$class.foreach(IterableLike.scala:72)
> at scala.collection.AbstractIterable.foreach(Iterable.scala:54)
> at
> kafka.log.Cleaner$$anonfun$kafka$log$Cleaner$$buildOffsetMapForSegment$1.apply(LogCleaner.scala:752)
> at
> kafka.log.Cleaner$$anonfun$kafka$log$Cleaner$$buildOffsetMapForSegment$1.apply(LogCleaner.scala:741)
> at scala.collection.Iterator$class.foreach(Iterator.scala:891)
> at scala.collection.AbstractIterator.foreach(Iterator.scala:1334)
> at scala.collection.IterableLike$class.foreach(IterableLike.scala:72)
> at scala.collection.AbstractIterable.foreach(Iterable.scala:54)
> at
> kafka.log.Cleaner.kafka$log$Cleaner$$buildOffsetMapForSegment(LogCleaner.scala:741)
> at
> kafka.log.Cleaner$$anonfun$buildOffsetMap$3.apply(LogCleaner.scala:707)
> at
> kafka.log.Cleaner$$anonfun$buildOffsetMap$3.apply(LogCleaner.scala:704)
> at
> scala.collection.TraversableLike$WithFilter$$anonfun$foreach$1.apply(TraversableLike.scala:733)
> at
> scala.collection.mutable.ResizableArray$class.foreach(ResizableArray.scala:59)
> at scala.collection.mutable.ArrayBuffer.foreach(ArrayBuffer.scala:48)
> at
> scala.collection.TraversableLike$WithFilter.foreach(TraversableLike.scala:732)
> at kafka.log.Cleaner.buildOffsetMap(LogCleaner.scala:704)
> at kafka.log.Cleaner.doClean(LogCleaner.scala:373)
> at kafka.log.Cleaner.clean(LogCleaner.scala:361)
> at
> kafka.log.LogCleaner$CleanerThread.cleanOrSleep(LogCleaner.scala:256)
> at kafka.log.LogCleaner$CleanerThread.doWork(LogCleaner.scala:236)
> at kafka.utils.ShutdownableThread.run(ShutdownableThread.scala:64)
> [2019-02-19 15:08:23,964] priority=INFO
> message="[kafka-log-cleaner-thread-0]: Stopped"
> category=kafka.log.LogCleaner{code}
> This is the first time we are seeing this error and coincidentally we have
> changed our JVM from Oracle to OpenJDK, both 1.8 Java.
>
> Please note that I can't upgrade to newer version of Kafka yet.
> Thanks!
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
