Thanks Tommi. I have again started documenting these changes in https://cwiki.apache.org/confluence/display/KAFKA/KIP-1030%3A+Change+constraints+and+default+values+for+various+configurations and will try to get this KIP out of draft stage next week (so that we can hit the 4.0 KIP freeze timeline of 20th Nov).
-- Divij Vaidya On Wed, Oct 30, 2024 at 12:46 PM Tommi Vainikainen <tvain...@aiven.io.invalid> wrote: > Hi, > > I've noticed that similar to these already mentioned settings also > segment.index.bytes has a minimum value of 4. This conflicts with > OffsetIndex, which throws `java.lang.IllegalArgumentException: Invalid max > index size: 4` with such settings, because hard-coded entry size in > OffsetIndex is 8. Setting segment.index.bytes to less than 8 leads to > errors. > > On Mon, Mar 11, 2024 at 7:33 PM Divij Vaidya <divijvaidy...@gmail.com> > wrote: > > > Hey folks > > > > Before I file a KIP to change this in 4.0, I wanted to understand the > > historical context for the value of the following setting. > > > > Currently, segment.ms minimum threshold is set to 1ms [1]. > > > > Segments are expensive. Every segment uses multiple file descriptors and > > it's easy to run out of OS limits when creating a large number of > segments. > > Large number of segments also delays log loading on startup because of > > expensive operations such as iterating through all directories & > > conditionally loading all producer state. > > > > I am currently not aware of a reason as to why someone might want to work > > with a segment.ms of less than ~10s (number chosen arbitrary that looks > > sane) > > > > What was the historical context of setting the minimum threshold to 1ms > for > > this setting? > > > > [1] https://kafka.apache.org/documentation.html#topicconfigs_segment.ms > > > > -- > > Divij Vaidya > > >
