Hi there, I have a kafka topic where my tombstone events don't get deleted and I don't know why. Maybe you can shed some light for me here?
First about the usage of that topic: In our application, we regularly create new entities and create a UUID for those. I publish these entitites as JSON to kafka with the UUID being the key. If a user deletes such an entity, I send a kafka tombstone event for this UUID (i.e. NULL payload. I can verify them being correct NULL tombstones via kafkacat %S option reporting -1 length as synonym for NULL payload). In the QA environment of the project, multiple times a day new entities are created and deleted. It's not big amounts of data but over the 1.5 years this topic exists, its about 300.000 messages in total. It's not an issue in production where entities stay for a long time, but it starts slowing down our QA environment on starting up some services consuming the entire topic.. The topic is configured with: cleanup.policy : compact All other properties are on default: retention.ms : 604800000 min.compaction.lag.ms : 0 max.compaction.lag.ms : 9223372036854775807 min.cleanable.dirty.ratio : 0.5 retention.bytes : -1 delete.retention.ms : 604800000 I see that never once a tombstone was deleted, but regular compaction runs: First offsets in the topic are deleted [e.g. 0] because they were updated with the tombstone, but the tombstone is still there (offset 3, 6, 8, ...). The first event is as old as our topic is (topic created end of december 2021. Plenty of time at least some tombstones could have been deleted...) $ docker run confluentinc/cp-kafkacat:latest kafkacat -b $BROKER -t $TOPIC -C -c10 -f '%o (%T) - %k: %S\n' 3 (1638790749420) - abf5a0eb-7838-4ae2-8487-d61c04568790: -1 6 (1638790754734) - 916fd2fd-a72c-4143-bf6e-12ca411df3d3: -1 8 (1638791287534) - 61413f83-58fe-489b-b132-9109bea1b28f: -1 10 (1638791302689) - 4617109e-2c18-4150-ae5b-e8ef1f805b99: -1 13 (1638791318536) - 88d2216e-522e-435d-9e94-9ee90490627a: -1 16 (1638791430823) - 0b9022a8-a565-44ea-9ac5-68e939e56aed: -1 18 (1638791466334) - f515ab0b-d535-4821-89ff-61ae18c7cea4: -1 21 (1638791648469) - 8f4b1a3b-3829-49de-ab9d-bdaee48dc360: -1 22 (1638791650116) - ae0d3fbb-3943-4a9f-93ad-923c5db93e79: -1 25 (1638791834467) - e55ec38e-a4db-4ed3-86a9-12ead58f8d21: -1 >From the filesystem perspective, in my opinion everything seems to be fine. >There are plenty of segments (one for each week) in that topic (The topic only >has one partition): $ ls -alh total 20M drwxr-xr-x 2 kafka kafka 4.0K Mar 22 17:06 . drwxr-xr-x 197 kafka kafka 12K Mar 27 08:41 .. -rw-r--r-- 1 kafka kafka 728 Jan 25 06:19 00000000000000000000.index -rw-r--r-- 1 kafka kafka 13M Jan 25 06:19 00000000000000000000.log -rw-r--r-- 1 kafka kafka 1.1K Jan 25 06:19 00000000000000000000.timeindex -rw-r--r-- 1 kafka kafka 1.5K Feb 1 11:13 00000000000000302362.index -rw-r--r-- 1 kafka kafka 760K Feb 1 10:53 00000000000000302362.log -rw-r--r-- 1 kafka kafka 2.2K Feb 1 11:13 00000000000000302362.timeindex -rw-r--r-- 1 kafka kafka 1.2K Feb 8 11:14 00000000000000307191.index -rw-r--r-- 1 kafka kafka 597K Feb 8 11:11 00000000000000307191.log -rw-r--r-- 1 kafka kafka 1.7K Feb 8 11:14 00000000000000307191.timeindex -rw-r--r-- 1 kafka kafka 2.0K Feb 15 12:00 00000000000000310913.index -rw-r--r-- 1 kafka kafka 1.1M Feb 15 11:00 00000000000000310913.log -rw-r--r-- 1 kafka kafka 3.1K Feb 15 12:00 00000000000000310913.timeindex -rw-r--r-- 1 kafka kafka 936 Feb 22 12:44 00000000000000316185.index -rw-r--r-- 1 kafka kafka 486K Feb 22 07:22 00000000000000316185.log -rw-r--r-- 1 kafka kafka 1.4K Feb 22 12:44 00000000000000316185.timeindex -rw-r--r-- 1 kafka kafka 1.6K Mar 1 17:03 00000000000000319076.index -rw-r--r-- 1 kafka kafka 803K Mar 1 05:42 00000000000000319076.log -rw-r--r-- 1 kafka kafka 2.3K Mar 1 17:03 00000000000000319076.timeindex -rw-r--r-- 1 kafka kafka 1.1K Mar 8 17:03 00000000000000323131.index -rw-r--r-- 1 kafka kafka 576K Mar 8 17:03 00000000000000323131.log -rw-r--r-- 1 kafka kafka 1.7K Mar 8 17:03 00000000000000323131.timeindex -rw-r--r-- 1 kafka kafka 1.3K Mar 15 17:04 00000000000000326243.index -rw-r--r-- 1 kafka kafka 661K Mar 15 17:03 00000000000000326243.log -rw-r--r-- 1 kafka kafka 1.9K Mar 15 17:04 00000000000000326243.timeindex -rw-r--r-- 1 kafka kafka 1.4K Mar 22 17:06 00000000000000329916.index -rw-r--r-- 1 kafka kafka 717K Mar 22 17:03 00000000000000329916.log -rw-r--r-- 1 kafka kafka 10 Mar 15 17:04 00000000000000329916.snapshot -rw-r--r-- 1 kafka kafka 2.1K Mar 22 17:06 00000000000000329916.timeindex -rw-r--r-- 1 kafka kafka 10M Mar 27 08:24 00000000000000333580.index -rw-r--r-- 1 kafka kafka 508K Mar 27 08:24 00000000000000333580.log -rw-r--r-- 1 kafka kafka 10 Mar 22 17:06 00000000000000333580.snapshot -rw-r--r-- 1 kafka kafka 10M Mar 27 08:24 00000000000000333580.timeindex -rw-r--r-- 1 kafka kafka 309 Feb 4 20:55 leader-epoch-checkpoint The kafka broker version we use is 2.5.0 (to be precise: 2.5.0.7.1.7.0-551 from Cloudera 7.1.7 cluster). Might this be related to [ https://issues.apache.org/jira/browse/KAFKA-8522 | https://issues.apache.org/jira/browse/KAFKA-8522 ] which was fixed in 3.1? But the assumption in that ticket is that this bug only applies where min.cleanable.dirty.ratio is rather small whereas for us, its still on the default on 0.5. Any other ideas why tombstones aren't removed in our case? Best regards Theo
