Badai, Thanks for the KIP.
We sometimes want to force compaction on a topic. This might be because there is a bad record in the topic, and we want to force it to get deleted. The way we do this is, we set segment.ms to a small value and write a record, in order to force a segment roll. And we also set min.cleanable.dirty.ratio=0, in order to trigger compaction. It's rare that we need to do it, but it happens sometimes. This change would make it more difficult to do that. With this KIP, we would have to write up to 1MB of data before causing the segment roll, or wait an hour. Although come to think of it, if my goal is to trigger compaction, then I can just write my tombstone a couple thousand times. So maybe this KIP just makes it slightly more tedious, but doesn't make it impossible. Another use case is when we want to truncate a topic, so we set a small segment size and set retention to almost zero, which will allow Kafka to delete what is in the topic. For that, though, we could also use kafka-delete-records.sh, so this KIP would not have impact on that particular use case. -James > On Jul 6, 2021, at 2:23 PM, Badai Aqrandista <ba...@confluent.io.INVALID> > wrote: > > Hi all > > I have just created KIP-760 > (https://cwiki.apache.org/confluence/display/KAFKA/KIP-760%3A+Increase+minimum+value+of+segment.ms+and+segment.bytes). > > I created this KIP because I have seen so many Kafka brokers crash due > to small segment.ms and/or segment.bytes. > > Please let me know what you think. > > -- > Thanks, > Badai
