James Thank you for replying. I originally thought about adding two new broker dynamic config to allow administrator to determine the acceptable minimum:
min.topic.segment.ms min.topic.segment.bytes I put this in the "rejected alternatives" section because I think adding two more configs to the broker would be too much trouble. Or is it not? Based on your use case, which is a valid use case, adding these dynamic config would still allow your use case to work, with an additional step of lowering the dynamic config temporarily. What do you think? Regards Badai On Wed, Jul 7, 2021 at 2:06 PM James Cheng <wushuja...@gmail.com> wrote: > > Badai, > > Thanks for the KIP. > > We sometimes want to force compaction on a topic. This might be because there > is a bad record in the topic, and we want to force it to get deleted. The way > we do this is, we set segment.ms to a small value and write a record, in > order to force a segment roll. And we also set min.cleanable.dirty.ratio=0, > in order to trigger compaction. It's rare that we need to do it, but it > happens sometimes. This change would make it more difficult to do that. With > this KIP, we would have to write up to 1MB of data before causing the segment > roll, or wait an hour. > > Although come to think of it, if my goal is to trigger compaction, then I can > just write my tombstone a couple thousand times. So maybe this KIP just makes > it slightly more tedious, but doesn't make it impossible. > > Another use case is when we want to truncate a topic, so we set a small > segment size and set retention to almost zero, which will allow Kafka to > delete what is in the topic. For that, though, we could also use > kafka-delete-records.sh, so this KIP would not have impact on that particular > use case. > > -James > > > On Jul 6, 2021, at 2:23 PM, Badai Aqrandista <ba...@confluent.io.INVALID> > > wrote: > > > > Hi all > > > > I have just created KIP-760 > > (https://cwiki.apache.org/confluence/display/KAFKA/KIP-760%3A+Increase+minimum+value+of+segment.ms+and+segment.bytes). > > > > I created this KIP because I have seen so many Kafka brokers crash due > > to small segment.ms and/or segment.bytes. > > > > Please let me know what you think. > > > > -- > > Thanks, > > Badai > -- Thanks, Badai
