Some further information, and is this a bug? I'm using 0.8.2.1. Log compaction will only occur on the non active segments. Intentional or not, it seems that the last segment is always the active segment. In other words, an expired segment will not be cleaned until a new segment has been created.
As a result, a log won't be compacted until new data comes in (per partition). Does this mean I need to send the equivalent of a pig ( https://en.wikipedia.org/wiki/Pigging) through each partition in order to force compaction? Or can I force the cleaning somehow? Here are the steps to recreate: 1. Create a new topic with a 5 minute segment.ms: kafka-topics.sh --zookeeper localhost:2181 --create --topic TEST_TOPIC --replication-factor 1 --partitions 1 --config cleanup.policy=compact --config min.cleanable.dirty.ratio=0.01 --config segment.ms=300000 2. Repeatedly add messages with identical keys (3x): echo "ABC123,{\"test\": 1}" | kafka-console-producer.sh --broker-list localhost:9092 --topic TEST_TOPIC --property parse.key=true --property key.separator=, --new-producer 3. Wait 5+ minutes and confirm no log compaction. 4. Once satisfied, send a new message: echo "DEF456,{\"test\": 1}" | kafka-console-producer.sh --broker-list localhost:9092 --topic TEST_TOPIC --property parse.key=true --property key.separator=, --new-producer 5. Log compaction will occur quickly soon after. Is my use case of infrequent logs not supported? Is this intentional behavior? It's unnecessarily challenging to target each partition with a dummy message to trigger compaction. Also, I believe there is another issue with logs originally configured without a segment timeout that lead to my original issue. I still cannot get those logs to compact. Thanks! Shayne
