[
https://issues.apache.org/jira/browse/KAFKA-3894?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15419166#comment-15419166
]
ASF GitHub Bot commented on KAFKA-3894:
---------------------------------------
GitHub user tcrayford opened a pull request:
https://github.com/apache/kafka/pull/1725
WIP KAFKA-3894: split log segment to avoid crashing cleaner thread
https://issues.apache.org/jira/browse/KAFKA-3894
This is a temporary PR, to see what Jenkins has to say about this work in
progress change. It will be updated and should not be reviewed at this time.
You can merge this pull request into a Git repository by running:
$ git pull https://github.com/heroku/kafka
dont_crash_log_cleaner_thread_if_segment_overflows_buffer
Alternatively you can review and apply these changes as the patch at:
https://github.com/apache/kafka/pull/1725.patch
To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:
This closes #1725
----
commit 995e32398c40c5d4deddeb9e5f359dc5df770b27
Author: Tom Crayford <tcrayf...@googlemail.com>
Date: 2016-08-12T17:20:26Z
WIP KAFKA-3894: split log segment to avoid crashing cleaner thread
----
> LogCleaner thread crashes if not even one segment can fit in the offset map
> ---------------------------------------------------------------------------
>
> Key: KAFKA-3894
> URL: https://issues.apache.org/jira/browse/KAFKA-3894
> Project: Kafka
> Issue Type: Bug
> Components: core
> Affects Versions: 0.8.2.2, 0.9.0.1, 0.10.0.0
> Environment: Oracle JDK 8
> Ubuntu Precise
> Reporter: Tim Carey-Smith
> Labels: compaction
> Fix For: 0.10.1.0
>
>
> The log-cleaner thread can crash if the number of keys in a topic grows to be
> too large to fit into the dedupe buffer.
> The result of this is a log line:
> {quote}
> broker=0 pri=ERROR t=kafka-log-cleaner-thread-0 at=LogCleaner
> \[kafka-log-cleaner-thread-0\], Error due to
> java.lang.IllegalArgumentException: requirement failed: 9750860 messages in
> segment MY_FAVORITE_TOPIC-2/00000000000047580165.log but offset map can fit
> only 5033164. You can increase log.cleaner.dedupe.buffer.size or decrease
> log.cleaner.threads
> {quote}
> As a result, the broker is left in a potentially dangerous situation where
> cleaning of compacted topics is not running.
> It is unclear if the broader strategy for the {{LogCleaner}} is the reason
> for this upper bound, or if this is a value which must be tuned for each
> specific use-case.
> Of more immediate concern is the fact that the thread crash is not visible
> via JMX or exposed as some form of service degradation.
> Some short-term remediations we have made are:
> * increasing the size of the dedupe buffer
> * monitoring the log-cleaner threads inside the JVM
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
