[ https://issues.apache.org/jira/browse/KAFKA-3894?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15365233#comment-15365233 ]
Tim Carey-Smith commented on KAFKA-3894: ---------------------------------------- Woohoo, more metrics is so excellent! Regarding the issue I am reporting: it is somewhat broader than the specific issues related to the log cleaner which have been resolved across the lifetime of Kafka. * Compaction thread dies when hitting compressed and unkeyed messages (https://github.com/apache/kafka/commit/1cd6ed9e2c07a63474ed80a8224bd431d5d4243c#diff-d7330411812d23e8a34889bee42fedfe) noted in KAFKA-1755 * Logcleaner fails due to incorrect offset map computation on a replica in KAFKA-3587 Unfortunately, there is a deeper issue: if these threads die, bad things happen. KAFKA-3587 was a great step forward, now this exception will only occur if a single segment is unable to fit within the dedupe buffer. Unfortunately, in pathological cases the thread could still die. Compacted topics are built to rely on the log cleaner thread and because of this, any segments which are written must be compatible with the configuration for log cleaner threads. As I mentioned before, we are now monitoring the log cleaner threads and as a result do not have long periods where a broker is in a dangerous and degraded state. One situation which comes to mind is from a talk at Kafka Summit where the thread was offline for a large period of time. Upon restart, the {{__consumer_offsets}} topic took 17 minutes to load. http://www.slideshare.net/jjkoshy/kafkaesque-days-at-linked-in-in-2015/49 After talking with Tom, we came up with a few solutions which could help in resolving this issue. 1) The monitoring suggested in KAFKA-3857 is a great start and would most definitely help with determining the state of the log cleaner. 2) After the change in KAFKA-3587, it could be possible to simply leave segments which are too large and leave them as zombie segments which will never be cleaned. This is less than ideal, but means that a single large segment would not take down the whole log cleaner subsystem. 3) Upon encountering a large segment, we considered the possibility of splitting the segment to allow the log cleaner to continue. This would potentially delay some cleanup until a later time. 4) Currently, it seems like the write path allows for segments to be created which are unable to be processed by the log cleaner. Would it make sense to include log cleaner heuristics when determining segment size for compacted topics? This would allow the log cleaner to always process a segment, unless the buffer size was changed. We'd love to help in any way we can. > Log Cleaner thread crashes and never restarts > --------------------------------------------- > > Key: KAFKA-3894 > URL: https://issues.apache.org/jira/browse/KAFKA-3894 > Project: Kafka > Issue Type: Bug > Components: core > Affects Versions: 0.8.2.2, 0.9.0.1 > Environment: Oracle JDK 8 > Ubuntu Precise > Reporter: Tim Carey-Smith > Labels: compaction > > The log-cleaner thread can crash if the number of keys in a topic grows to be > too large to fit into the dedupe buffer. > The result of this is a log line: > {quote} > broker=0 pri=ERROR t=kafka-log-cleaner-thread-0 at=LogCleaner > \[kafka-log-cleaner-thread-0\], Error due to > java.lang.IllegalArgumentException: requirement failed: 9750860 messages in > segment MY_FAVORITE_TOPIC-2/00000000000047580165.log but offset map can fit > only 5033164. You can increase log.cleaner.dedupe.buffer.size or decrease > log.cleaner.threads > {quote} > As a result, the broker is left in a potentially dangerous situation where > cleaning of compacted topics is not running. > It is unclear if the broader strategy for the {{LogCleaner}} is the reason > for this upper bound, or if this is a value which must be tuned for each > specific use-case. > Of more immediate concern is the fact that the thread crash is not visible > via JMX or exposed as some form of service degradation. > Some short-term remediations we have made are: > * increasing the size of the dedupe buffer > * monitoring the log-cleaner threads inside the JVM -- This message was sent by Atlassian JIRA (v6.3.4#6332)
