[
https://issues.apache.org/jira/browse/KAFKA-3894?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15382370#comment-15382370
]
Vincent Rischmann commented on KAFKA-3894:
------------------------------------------
Not adding much to the conversation, but I've just been hit by this bug.
I'm in the process of upgrading my cluster to 0.9.0.1, and in one case the log
cleaner dies because of this.
{{requirement failed: 1214976153 messages in segment
__consumer_offsets-15/00000000000012560043.log but offset map can fit only
40265317}}
If I'm not wrong, there's no way that much messages can fit in the buffer since
it's limited to 2G anyway per thread. Right now I'm leaving it as is since the
broker seems to be working, but it's not ideal.
I'm wondering if I simply delete the log file with the broker shut down, will
it be fetched at startup from an other replica without problems ?
In my case, I believe this is only temporary: we never enabled the log cleaner
when running 0.8.2.1 (mistake on my part) and now when migrating to 0.9.0.1 it
does a giant cleanup at first startup.
> Log Cleaner thread crashes and never restarts
> ---------------------------------------------
>
> Key: KAFKA-3894
> URL: https://issues.apache.org/jira/browse/KAFKA-3894
> Project: Kafka
> Issue Type: Bug
> Components: core
> Affects Versions: 0.8.2.2, 0.9.0.1
> Environment: Oracle JDK 8
> Ubuntu Precise
> Reporter: Tim Carey-Smith
> Labels: compaction
>
> The log-cleaner thread can crash if the number of keys in a topic grows to be
> too large to fit into the dedupe buffer.
> The result of this is a log line:
> {quote}
> broker=0 pri=ERROR t=kafka-log-cleaner-thread-0 at=LogCleaner
> \[kafka-log-cleaner-thread-0\], Error due to
> java.lang.IllegalArgumentException: requirement failed: 9750860 messages in
> segment MY_FAVORITE_TOPIC-2/00000000000047580165.log but offset map can fit
> only 5033164. You can increase log.cleaner.dedupe.buffer.size or decrease
> log.cleaner.threads
> {quote}
> As a result, the broker is left in a potentially dangerous situation where
> cleaning of compacted topics is not running.
> It is unclear if the broader strategy for the {{LogCleaner}} is the reason
> for this upper bound, or if this is a value which must be tuned for each
> specific use-case.
> Of more immediate concern is the fact that the thread crash is not visible
> via JMX or exposed as some form of service degradation.
> Some short-term remediations we have made are:
> * increasing the size of the dedupe buffer
> * monitoring the log-cleaner threads inside the JVM
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
