Thanks for the details Rajini. It would be great if you can add a few details to the KIP, on how many connections you are able to handle in your cluster with number 20 to give some context.
Thanks, Harsha On Tue, Dec 11, 2018, at 10:22 AM, Rajini Sivaram wrote: > Hi Harsha, > > Thanks for reviewing the KIP. > > 1) Yes, agree that we also need a max.connections configuration per-broker. > I was thinking of doing that in a separate KIP, but I could add that here > as well. > 2) The number of connections processed in each iteration doesn't feel like > an externalizable config.It is not a limit on connection rate, it is simply > ensuring that existing connections are processed by each Processor after > atmost every 20 new connections. It will be hard to describe this > configuration for users to enable configuring this in a way that is > suitable for a connection flood since it would depend on the number of > factors like existing connection count etc. It feels like we should come up > with a number that works well. We have been running with this code for a > while and so far haven't run into any noticeable degradations with 20. > > > > On Tue, Dec 11, 2018 at 6:03 PM Harsha <ka...@harsha.io> wrote: > > > Hi Rajini, > > Overall KIP looks good to me. Is it possible to use > > max.connections config that we already have, althought its per IP. > > But broker level max.connections would also be good have to guard against > > DOS'ing a broker. > > Eitherway having constant like 20 without a configurable option doesn't > > sound right and as the KIP states that one can use num.network.threads to > > increase this capacity, it still not a viable option. Most of the time > > users tend to keep network threads minimal and given this configuration > > will only need when a burst of requests comes through , allowing users to > > choose that ceiling would be beneficial. Can you add any details on why 20 > > is sufficient , with default num.network.threads with 3 if one broker is > > getting more than 60 simultaneous connections this would result in > > perceived slower responses from client side right? > > > > Thanks, > > Harsha > > > > > > On Tue, Dec 11, 2018, at 2:48 AM, Rajini Sivaram wrote: > > > Hi all, > > > > > > I have submitted a KIP to improve fairness in channel processing in > > > SocketServer to protect brokers from connection storms: > > > > > > - > > > > > > > > https://cwiki.apache.org/confluence/display/KAFKA/KIP-402%3A+Improve+fairness+in+SocketServer+processors > > > > > > Feedback and suggestions welcome. > > > > > > Thank you, > > > > > > Rajini > >
