[
https://issues.apache.org/jira/browse/KAFKA-7951?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16773631#comment-16773631
]
huxihx edited comment on KAFKA-7951 at 2/21/19 3:26 AM:
--------------------------------------------------------
[~ninth.nails] Had you upgraded or downgraded the Kafka version before running
into this problem? The stack trace showed the exception was thrown when reading
the record size. The value should not be that large.
was (Author: huxi_2b):
[~ninth.nails] Had you upgraded or downgraded the Kafka version before running
into this problem? The stack trace showed the exception was thrown when reading
the record size. That value should not be very large.
> Log Cleaner thread stop with "Varint is too long" error
> -------------------------------------------------------
>
> Key: KAFKA-7951
> URL: https://issues.apache.org/jira/browse/KAFKA-7951
> Project: Kafka
> Issue Type: Bug
> Components: log
> Affects Versions: 0.11.0.3
> Environment: Amazon Linux AMI 2017.09
> confluent-platform-oss-2.11-3.3.2
> OpenJDK Runtime Environment (build 1.8.0_191-b12)
> EC2 instance type i3.xlarge
> Reporter: Ninth Nails
> Priority: Blocker
>
> I have one broker log cleaner thread dying because of a failing type
> conversion. My cluster is running on 0.11.03-cp1 (Confluent Platform). Here's
> the relevant logs in log-cleaner.log
> {code:java}
> [2019-02-19 15:08:23,387] priority=INFO message="[kafka-log-cleaner-thread-0]:
> Log cleaner thread 0 cleaned log __consumer_offsets-36 (dirty section
> = [931148954, 931148954])
> 13,051.5 MB of log processed in 159.0 seconds (82.1 MB/sec).
> Indexed 13,051.5 MB in 81.2 seconds (160.8 Mb/sec, 51.1% of total
> time)
> Buffer utilization: 0.0%
> Cleaned 13,051.5 MB in 77.8 seconds (167.7 Mb/sec, 48.9% of total
> time)
> Start size: 13,051.5 MB (166,676,749 messages)
> End size: 0.0 MB (159 messages)
> 100.0% size reduction (100.0% fewer messages)
> " category=kafka.log.LogCleaner
> [2019-02-19 15:08:23,392] priority=INFO message="Cleaner 0: Beginning
> cleaning of log __consumer_offsets-37." category=kafka.log.LogCleaner
> [2019-02-19 15:08:23,392] priority=INFO message="Cleaner 0: Building offset
> map for __consumer_offsets-37..." category=kafka.log.LogCleaner
> [2019-02-19 15:08:23,410] priority=INFO message="Cleaner 0: Building offset
> map for log __consumer_offsets-37 for 87 segments in offset range [211566448,
> 308245257)." category=kafka.log.LogCleaner
> [2019-02-19 15:08:23,945] priority=ERROR
> message="[kafka-log-cleaner-thread-0]: Error due to"
> category=kafka.log.LogCleaner
> java.lang.IllegalArgumentException: Varint is too long, the most significant
> bit in the 5th byte is set, converted value: 445d418c
> at
> org.apache.kafka.common.utils.ByteUtils.illegalVarintException(ByteUtils.java:326)
> at
> org.apache.kafka.common.utils.ByteUtils.readVarint(ByteUtils.java:148)
> at
> org.apache.kafka.common.record.DefaultRecord.readFrom(DefaultRecord.java:305)
> at
> org.apache.kafka.common.record.DefaultRecordBatch$2.readNext(DefaultRecordBatch.java:299)
> at
> org.apache.kafka.common.record.DefaultRecordBatch$RecordIterator.next(DefaultRecordBatch.java:563)
> at
> org.apache.kafka.common.record.DefaultRecordBatch$RecordIterator.next(DefaultRecordBatch.java:532)
> at
> scala.collection.convert.Wrappers$JIteratorWrapper.next(Wrappers.scala:43)
> at scala.collection.Iterator$class.foreach(Iterator.scala:891)
> at scala.collection.AbstractIterator.foreach(Iterator.scala:1334)
> at scala.collection.IterableLike$class.foreach(IterableLike.scala:72)
> at scala.collection.AbstractIterable.foreach(Iterable.scala:54)
> at
> kafka.log.Cleaner$$anonfun$kafka$log$Cleaner$$buildOffsetMapForSegment$1.apply(LogCleaner.scala:752)
> at
> kafka.log.Cleaner$$anonfun$kafka$log$Cleaner$$buildOffsetMapForSegment$1.apply(LogCleaner.scala:741)
> at scala.collection.Iterator$class.foreach(Iterator.scala:891)
> at scala.collection.AbstractIterator.foreach(Iterator.scala:1334)
> at scala.collection.IterableLike$class.foreach(IterableLike.scala:72)
> at scala.collection.AbstractIterable.foreach(Iterable.scala:54)
> at
> kafka.log.Cleaner.kafka$log$Cleaner$$buildOffsetMapForSegment(LogCleaner.scala:741)
> at
> kafka.log.Cleaner$$anonfun$buildOffsetMap$3.apply(LogCleaner.scala:707)
> at
> kafka.log.Cleaner$$anonfun$buildOffsetMap$3.apply(LogCleaner.scala:704)
> at
> scala.collection.TraversableLike$WithFilter$$anonfun$foreach$1.apply(TraversableLike.scala:733)
> at
> scala.collection.mutable.ResizableArray$class.foreach(ResizableArray.scala:59)
> at scala.collection.mutable.ArrayBuffer.foreach(ArrayBuffer.scala:48)
> at
> scala.collection.TraversableLike$WithFilter.foreach(TraversableLike.scala:732)
> at kafka.log.Cleaner.buildOffsetMap(LogCleaner.scala:704)
> at kafka.log.Cleaner.doClean(LogCleaner.scala:373)
> at kafka.log.Cleaner.clean(LogCleaner.scala:361)
> at
> kafka.log.LogCleaner$CleanerThread.cleanOrSleep(LogCleaner.scala:256)
> at kafka.log.LogCleaner$CleanerThread.doWork(LogCleaner.scala:236)
> at kafka.utils.ShutdownableThread.run(ShutdownableThread.scala:64)
> [2019-02-19 15:08:23,964] priority=INFO
> message="[kafka-log-cleaner-thread-0]: Stopped"
> category=kafka.log.LogCleaner{code}
> This is the first time we are seeing this error and coincidentally we have
> changed our JVM from Oracle to OpenJDK, both 1.8 Java.
>
> Please note that I can't upgrade to newer version of Kafka yet.
> Thanks!
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
