I understand Kafka supports keyed messages (I am using 0.8.1.1) and it is
possible to de-duplicate messages based on the message key.
(The log compaction section of the on-line documentation described how that
works.)
I am using a code example that come with Kafka (namely
KafkaConsumerProducerDem
It looks like the log.cleanup.policy config option was changed from "dedupe" to
"compact".
https://github.com/apache/kafka/blob/0.8.1.1/core/src/main/scala/kafka/log/LogConfig.scala#L68
-Jon
On May 13, 2014, at 1:08 PM, Jay Kreps wrote:
> Hi,
>
> The compaction is done to clean-up space. It
Hi,
The compaction is done to clean-up space. It isn't done immediately only
periodically.
I suspect the reason you see no compaction is that we never compact the
active segment of the log (the most recent file) as that is still being
written to. The compaction would not happen until a new segmen
