Hi all, There are also message.max.bytes, replica.fetch.max.bytes and their derivatives requires a constraint on their maximum value as the maximum total memory on the instance. Otherwise, these could cause out of memory errors on the instance.
Do you think this is in scope here as well? On Thu, 14 Mar 2024 at 10:29, Haruki Okada <ocadar...@gmail.com> wrote: > Hi, Divij. > > This isn't about config default value/constraint change though, I found > there's a behavior discrepancy in max.block.ms config, which may cause > breaking change if we change the behavior. > The detail is described in the ticket: > https://issues.apache.org/jira/browse/KAFKA-16372 > > What do you think? > > 2024年3月14日(木) 13:09 Kamal Chandraprakash <kamal.chandraprak...@gmail.com>: > > > One use case I see for setting the `segment.bytes` to 1 is to delete all > > the records from the topic. > > We can mention about it in the doc to use the `kafka-delete-records` API > > instead. > > > > > > > > > > On Wed, Mar 13, 2024 at 6:59 PM Divij Vaidya <divijvaidy...@gmail.com> > > wrote: > > > > > + users@kafka > > > > > > Hi users of Apache Kafka > > > > > > With the upcoming 4.0 release, we have an opportunity to improve the > > > constraints and default values for various Kafka configurations. > > > > > > We are soliciting your feedback and suggestions on configurations where > > the > > > default values and/or constraints should be adjusted. Please reply in > > this > > > thread directly. > > > > > > -- > > > Divij Vaidya > > > Apache Kafka PMC > > > > > > > > > > > > On Wed, Mar 13, 2024 at 12:56 PM Divij Vaidya <divijvaidy...@gmail.com > > > > > wrote: > > > > > > > Thanks for the discussion folks. I have started a KIP > > > > > > > > > > https://cwiki.apache.org/confluence/display/KAFKA/KIP-1030%3A+Change+constraints+and+default+values+for+various+configurations > > > > to keep track of the changes that we are discussion. Please consider > > this > > > > as a collaborative work-in-progress KIP and once it is ready to be > > > > published, we can start a discussion thread on it. > > > > > > > > I am also going to start a thread to solicit feedback from users@ > > > mailing > > > > list as well. > > > > > > > > -- > > > > Divij Vaidya > > > > > > > > > > > > > > > > On Wed, Mar 13, 2024 at 12:55 PM Christopher Shannon < > > > > christopher.l.shan...@gmail.com> wrote: > > > > > > > >> I think it's a great idea to raise a KIP to look at adjusting > defaults > > > and > > > >> minimum/maximum config values for version 4.0. > > > >> > > > >> As pointed out, the minimum values for segment.ms and segment.bytes > > > don't > > > >> make sense and would probably bring down a cluster pretty quickly if > > set > > > >> that low, so version 4.0 is a good time to fix it and to also look > at > > > the > > > >> other configs as well for adjustments. > > > >> > > > >> On Wed, Mar 13, 2024 at 4:39 AM Sergio Daniel Troiano > > > >> <sergio.troi...@adevinta.com.invalid> wrote: > > > >> > > > >> > hey guys, > > > >> > > > > >> > Regarding to num.recovery.threads.per.data.dir: I agree, in our > > > company > > > >> we > > > >> > use the number of vCPUs to do so as this is not competing with > ready > > > >> > cluster traffic. > > > >> > > > > >> > > > > >> > On Wed, 13 Mar 2024 at 09:29, Luke Chen <show...@gmail.com> > wrote: > > > >> > > > > >> > > Hi Divij, > > > >> > > > > > >> > > Thanks for raising this. > > > >> > > The valid minimum value 1 for `segment.ms` is completely > > > >> unreasonable. > > > >> > > Similarly for `segment.bytes`, `metadata.log.segment.ms`, > > > >> > > `metadata.log.segment.bytes`. > > > >> > > > > > >> > > In addition to that, there are also some config default values > > we'd > > > >> like > > > >> > to > > > >> > > propose to change in v4.0. > > > >> > > We can collect more comments from the community, and come out > > with a > > > >> KIP > > > >> > > for them. > > > >> > > > > > >> > > 1. num.recovery.threads.per.data.dir: > > > >> > > The current default value is 1. But the log recovery is > happening > > > >> before > > > >> > > brokers are in ready state, which means, we should use all the > > > >> available > > > >> > > resource to speed up the log recovery to bring the broker to > ready > > > >> state > > > >> > > soon. Default value should be... maybe 4 (to be decided)? > > > >> > > > > > >> > > 2. Other configs might be able to consider to change the > default, > > > but > > > >> > open > > > >> > > for comments: > > > >> > > 2.1. num.replica.fetchers: default is 1, but that's not > enough > > > when > > > >> > > there are multiple partitions in the cluster > > > >> > > 2.2. > `socket.send.buffer.bytes`/`socket.receive.buffer.bytes`: > > > >> > > Currently, we set 100kb as default value, but that's not enough > > for > > > >> > > high-speed network. > > > >> > > > > > >> > > Thank you. > > > >> > > Luke > > > >> > > > > > >> > > > > > >> > > On Tue, Mar 12, 2024 at 1:32 AM Divij Vaidya < > > > divijvaidy...@gmail.com > > > >> > > > > >> > > wrote: > > > >> > > > > > >> > > > Hey folks > > > >> > > > > > > >> > > > Before I file a KIP to change this in 4.0, I wanted to > > understand > > > >> the > > > >> > > > historical context for the value of the following setting. > > > >> > > > > > > >> > > > Currently, segment.ms minimum threshold is set to 1ms [1]. > > > >> > > > > > > >> > > > Segments are expensive. Every segment uses multiple file > > > descriptors > > > >> > and > > > >> > > > it's easy to run out of OS limits when creating a large number > > of > > > >> > > segments. > > > >> > > > Large number of segments also delays log loading on startup > > > because > > > >> of > > > >> > > > expensive operations such as iterating through all > directories & > > > >> > > > conditionally loading all producer state. > > > >> > > > > > > >> > > > I am currently not aware of a reason as to why someone might > > want > > > to > > > >> > work > > > >> > > > with a segment.ms of less than ~10s (number chosen arbitrary > > that > > > >> > looks > > > >> > > > sane) > > > >> > > > > > > >> > > > What was the historical context of setting the minimum > threshold > > > to > > > >> 1ms > > > >> > > for > > > >> > > > this setting? > > > >> > > > > > > >> > > > [1] > > > >> > > https://kafka.apache.org/documentation.html#topicconfigs_segment.ms > > > >> > > > > > > >> > > > -- > > > >> > > > Divij Vaidya > > > >> > > > > > > >> > > > > > >> > > > > >> > > > > > > > > > > > > -- > ======================== > Okada Haruki > ocadar...@gmail.com > ======================== >
