log.retention.bytes is a broker-level config that sets the maximum size of a topic partition on a broker, so it will apply to all topics…
unless a topic has the retention.bytes property configured — this is a topic-level config and only applies to a single topic — in which case that takes precedence. Kafka does not have a built-in mechanism for preventing full disks. You must do some topic growth prediction and set your topic retentions to be proportional to the total storage available per broker. As with most times disks fill up, it’s not enjoyable dealing with the fallout, so spending the time to get it right is well worth it. -- Peter (from phone) > On Feb 24, 2021, at 7:46 AM, Calvin Chen <pingc...@hotmail.com> wrote: > > Hi all > > I have question about Kafka topic log retention on bytes, does the > log.retention.bytes apply to each topic or it apply to all topics in broker? > If it apply to each topic, then, when topic numbers keep growing in broker, > how can we make sure total disk size for all topic logs will not exceed total > disk capacity of that broker? > > Thanks > Calvin >
