It looks like the log.cleanup.policy config option was changed from "dedupe" to "compact".
https://github.com/apache/kafka/blob/0.8.1.1/core/src/main/scala/kafka/log/LogConfig.scala#L68 -Jon On May 13, 2014, at 1:08 PM, Jay Kreps <jay.kr...@gmail.com> wrote: > Hi, > > The compaction is done to clean-up space. It isn't done immediately only > periodically. > > I suspect the reason you see no compaction is that we never compact the > active segment of the log (the most recent file) as that is still being > written to. The compaction would not happen until a new segment file was > rolled. If you want to see this happen I recommend changing the file > segment size configuration to something small (5mb) and produce enough > messages to roll a new segment file. You should then see logging about > compaction in logs/log-cleaner.log. > > -Jay > > > On Tue, May 13, 2014 at 11:52 AM, C 4.5 <cfourf...@gmail.com> wrote: > >> I understand Kafka supports keyed messages (I am using 0.8.1.1) and it is >> possible to de-duplicate messages based on the message key. >> >> (The log compaction section of the on-line documentation described how that >> works.) >> >> I am using a code example that come with Kafka (namely >> KafkaConsumerProducerDemo) and run it through Kafka local mode. I write a >> set of messages with the same String key and then have a consumer that >> consumes data. >> >> The consumer consumes messages *only* after the producer has produced all >> its messages. >> >> I would expect the consumer to retrieve only the latest message (as all >> messages have the same key) but it retrieves all messages the producer has >> emitted. >> >> I have also turned on these properties in the Kafka server: >> >> log.cleaner.enable=true >> log.cleanup.policy=dedupe >> >> - is de-duplication of messages guaranteed to take effect only after >> compaction? >> >> - I have tried to "force" compaction by setting "log.cleaner.backoff.ms" >> and "log.cleaner.min.cleanabke.ratio" to very low values, but I still >> observe the same behavior. >> >> Any ideas or pointers? >> >> Thanks. >>
signature.asc
Description: Message signed with OpenPGP using GPGMail
