Setting topic policy to "compact,delete" should be sufficient. Cf. https://cwiki.apache.org/confluence/display/KAFKA/KIP-71%3A+Enable+log+compaction+and+deletion+to+co-exist
Note: retention time is not based on wall-clock time, but embedded record timestamps. Thus, old messages get only deleted if new messages with larger timestamps are written to the topic. -Matthias On 10/6/17 6:17 AM, Asko Alhoniemi wrote: > Hello > > I understand that the compacted topic is meant to keep at least the latest > key value pair. > > However, I am having an issue since it can happen that entry becomes old > and I need to remove it. It may also occur, that I am not able to send key > "null" pair. So I need another method to remove my hanging entries. My hope > was with the following configuration: > > Topic:test.topic PartitionCount:30 ReplicationFactor:2 > Configs:segment.bytes=1048576,min.cleanable.dirty.ratio=0.1, > delete.retention.ms=1800000,retention.ms=900000,segment.ms > =900000,cleanup.policy=compact,delete > Topic: test.topic Partition: 0 Leader: 0 Replicas: 1,0 Isr: > 0 > Topic: test.topic Partition: 1 Leader: 0 Replicas: 0,1 Isr: > 0 > Topic: test.topic Partition: 2 Leader: 0 Replicas: 1,0 Isr: > 0 > Topic: test.topic Partition: 3 Leader: 0 Replicas: 0,1 Isr: > 0 > Topic: test.topic Partition: 4 Leader: 0 Replicas: 1,0 Isr: > 0 > Topic: test.topic Partition: 5 Leader: 0 Replicas: 0,1 Isr: > 0 > Topic: test.topic Partition: 6 Leader: 0 Replicas: 1,0 Isr: > 0 > Topic: test.topic Partition: 7 Leader: 0 Replicas: 0,1 Isr: > 0 > Topic: test.topic Partition: 8 Leader: 0 Replicas: 1,0 Isr: > 0 > Topic: test.topic Partition: 9 Leader: 0 Replicas: 0,1 Isr: > 0 > Topic: test.topic Partition: 10 Leader: 0 Replicas: 1,0 > Isr: 0 > Topic: test.topic Partition: 11 Leader: 0 Replicas: 0,1 > Isr: 0 > Topic: test.topic Partition: 12 Leader: 0 Replicas: 1,0 > Isr: 0 > Topic: test.topic Partition: 13 Leader: 0 Replicas: 0,1 > Isr: 0 > Topic: test.topic Partition: 14 Leader: 0 Replicas: 1,0 > Isr: 0 > Topic: test.topic Partition: 15 Leader: 0 Replicas: 0,1 > Isr: 0 > Topic: test.topic Partition: 16 Leader: 0 Replicas: 1,0 > Isr: 0 > Topic: test.topic Partition: 17 Leader: 0 Replicas: 0,1 > Isr: 0 > Topic: test.topic Partition: 18 Leader: 0 Replicas: 1,0 > Isr: 0 > Topic: test.topic Partition: 19 Leader: 0 Replicas: 0,1 > Isr: 0 > Topic: test.topic Partition: 20 Leader: 0 Replicas: 1,0 > Isr: 0 > Topic: test.topic Partition: 21 Leader: 0 Replicas: 0,1 > Isr: 0 > Topic: test.topic Partition: 22 Leader: 0 Replicas: 1,0 > Isr: 0 > Topic: test.topic Partition: 23 Leader: 0 Replicas: 0,1 > Isr: 0 > Topic: test.topic Partition: 24 Leader: 0 Replicas: 1,0 > Isr: 0 > Topic: test.topic Partition: 25 Leader: 0 Replicas: 0,1 > Isr: 0 > Topic: test.topic Partition: 26 Leader: 0 Replicas: 1,0 > Isr: 0 > Topic: test.topic Partition: 27 Leader: 0 Replicas: 0,1 > Isr: 0 > Topic: test.topic Partition: 28 Leader: 0 Replicas: 1,0 > Isr: 0 > Topic: test.topic Partition: 29 Leader: 0 Replicas: 0,1 > Isr: 0 > > but it does not get cleaned. I stop writing one key value pair, but still > can see that key value pair after days. > > Then I changed broker configuration on the fly: > > # The minimum age of a log file to be eligible for deletion > #log.retention.hours=168 > log.retention.minutes=16 > > # A size-based retention policy for logs. Segments are pruned from the log > as long as the remaining > # segments don't drop below log.retention.bytes. > #log.retention.bytes=1073741824 > log.retention.bytes=10000 > > # The maximum size of a log segment file. When this size is reached a new > log segment will be created. > #log.segment.bytes=1073741824 > log.segment.bytes=10000 > > So far I haven't been able to drop away the desired key value pair with any > of these configuration actions. > > Am I trying something that is not possible or am I missing some > configuration options or should this work? > > br. > > Asko Alhoniemi >
signature.asc
Description: OpenPGP digital signature
