Hi Hector, I did see your colleague's KIP, and I actually mentioned it in the KIP that I have written. As I see it, both of these KIPs move towards more easily configurable replication throttling and both should be implemented. KIP-1009 makes it easier to enable throttling and KIP-1051 makes it easier to apply a throttle rate. I did try to look at supporting KIP-1009 in the discussion thread, however, I only subscribed to the mailing list after it was published and I couldn't figure out how to respond to it in Pony mail. I would be definitely be interested in partnering up to get both changes across the line, whether that be by combining them or supporting both individually (I'm not sure which is best, this is my first contribution!).
I also see that KAFKA-10190 is mentioned in KIP-1009 as a related ticket. Coincidentally, I raised a PR to address this bug a couple of days ago (https://github.com/apache/kafka/pull/16280). I think this is also a change that will move towards more easily configurable replication throttling as it allows configuring the throttle rate across the whole cluster via a default value. As far as I understand, this change does not need a KIP though because it is a bugfix (the current behaviour of ignoring the default is unintentional). Let me know what you think. Kind regards, Harry -------- Original Message -------- On 6/13/24 19:08, Hector Geraldino (BLOOMBERG/ 919 3RD A) <hgerald...@bloomberg.net> wrote: > Hi Harry, > > A colleague of mine opened KIP-1009: Add Broker-level Throttle > Configurations, which aims to achieve the same goal (although from a > different angle). > > Can you please take a look and see if this would work for the things you > have in mind? Maybe we can partner and coalesce around either KIP and try to > push it to the end line. > > KIP: > https://cwiki.apache.org/confluence/display/KAFKA/KIP-1009%3A+Add+Broker-level+Throttle+Configurations > > From: dev@kafka.apache.org At: 06/13/24 09:22:40 UTC-4:00To: > dev@kafka.apache.org > Subject: Re: [DISCUSS] KIP-1051 Statically configured log replication > throttling > > Hi everyone, > > Bumping this thread, as I haven't yet had any replies. > > Kind regards, > Harry > > On Thursday, 6 June 2024 at 17:59, Harry Fallows > <harryfall...@protonmail.com.INVALID> wrote: > > > Hi everyone, > > > > I would like to propose a change to allow the static configuration of > leader > and follower replication throttling rates. > > > > These configurations are very useful for preventing client traffic from > getting throttled by replication traffic during events that cause a spike in > replication. Currently they are only configurable dynamically, which means > they > are only really useful for throttling replication traffic during planned > events. By allowing these configurations to be set statically, they can be > used > to prevent client traffic throttling during unplanned events. > > > > KIP: > > https://cwiki.apache.org/confluence/display/KAFKA/KIP-1051%3A+Statically+configu > red+log+replication+throttling > > > > Best regards, > > Harry Fallows > > >
