[
https://issues.apache.org/jira/browse/KAFKA-3894?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15376858#comment-15376858
]
Tom Crayford commented on KAFKA-3894:
-------------------------------------
Jun:
#4 seems potentially very complex to me. It also doesn't work in the case that
the broker is shut down and the dedupe buffer size adjusted. I much prefer #3 -
it maps fine into the existing model as far as I can tell - we'd "just" split
the log file we're cleaning once the offsetmap is full. That of course requires
a little more IO, but it doesn't involve implementing (or using a library for)
sketches that could potentially be incorrect. It also seems like the right long
term solution, and more robust than automatically rolling log files some of the
time. Am I missing something here?
Upsides of #3 vs #4:
We can now clean the largest log segment, no matter the buffer size.
We don't increase complexity of the produce path, or change memory usage.
We don't have to implement or reuse a library for estimating unique keys
We don't have to figure out storing the key estimate (e.g. in the index or in a
new file alongside each segment).
Downsides:
It would increase the complexity of the cleaner.
The code that swaps in and out segments will also get more complex, and the
crash-safety of that code is already tricky.
Exists in both:
Larger log segments could potentially be split a lot, and not always
deduplicated that well together. For example, if I write the max number of
unique keys for the offset map into a topic, then the segment rolls, then I
write a tombstone for every message in the previously sent messages, then
neither #3 nor #4 would ever clear up any data. This is no worse than today
though.
Cassandra and other LSM based systems that do log structured storage and
over-time compaction use similar "splitting and combining" mechanisms to ensure
everything gets cleared up over time without using too much memory. They have a
very different storage architecture and goals to Kafka's compaction, for sure,
but it's interesting to note that they care about similar things.
> Log Cleaner thread crashes and never restarts
> ---------------------------------------------
>
> Key: KAFKA-3894
> URL: https://issues.apache.org/jira/browse/KAFKA-3894
> Project: Kafka
> Issue Type: Bug
> Components: core
> Affects Versions: 0.8.2.2, 0.9.0.1
> Environment: Oracle JDK 8
> Ubuntu Precise
> Reporter: Tim Carey-Smith
> Labels: compaction
>
> The log-cleaner thread can crash if the number of keys in a topic grows to be
> too large to fit into the dedupe buffer.
> The result of this is a log line:
> {quote}
> broker=0 pri=ERROR t=kafka-log-cleaner-thread-0 at=LogCleaner
> \[kafka-log-cleaner-thread-0\], Error due to
> java.lang.IllegalArgumentException: requirement failed: 9750860 messages in
> segment MY_FAVORITE_TOPIC-2/00000000000047580165.log but offset map can fit
> only 5033164. You can increase log.cleaner.dedupe.buffer.size or decrease
> log.cleaner.threads
> {quote}
> As a result, the broker is left in a potentially dangerous situation where
> cleaning of compacted topics is not running.
> It is unclear if the broader strategy for the {{LogCleaner}} is the reason
> for this upper bound, or if this is a value which must be tuned for each
> specific use-case.
> Of more immediate concern is the fact that the thread crash is not visible
> via JMX or exposed as some form of service degradation.
> Some short-term remediations we have made are:
> * increasing the size of the dedupe buffer
> * monitoring the log-cleaner threads inside the JVM
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
