Hello,
I'm using Kafka 0.9 and have a topic with the
config: cleanup.policy=compact, delete.retention.ms=30000, segment.ms
=30000, min.cleanable.dirty.ratio=0.01.
I understood regarding the requirements with the latest segments and how
only the segments other than the latest (active) are compacted.
Imagine I have the messages-(key,value) on segment 1 as:
key1 - msg1
key1 - msg2
Then the log rolls over according to segment.ms and creates a new segment
and I insert the value
key1 - msg3
I want to see only one message finally (key1, msg3) but when I consume from
oldest offset I get two messages msg2 and msg3 with the same key1. And I
see that log-cleaner.log has only the log displayed as :
Start size: 0.0 MB (2 messages)
End size: 0.0 MB (1 messages),
which corresponds to the first segment being compacted. I get the reason
behind this is that the new segment is active and is not compacted.
But is there any work around to make sure that I only see (key1,msg3) after
these operations
One buggy way I can think of is inserting a dummy (key2,msg) after
segment.ms duration, which will compact the key1 messages in the old
segments and give me two messages: (key1, msg3) and (key2, msg). But I do
not want this new key2 message :(
Any help is greatly appreciated.
Thank You
Ajay
