Hi Andrei, Kafka intentionally sets TCP_NODELAY in order to disable Nagle's algorithm. The reason is because Nagle's algorithm can cause to long latencies in Kafka RPC.
Wikipedia explains the problem: > [Nagle's] algorithm interacts badly with TCP delayed acknowledgments > (delayed ACK), a feature introduced into TCP at roughly the same time in > the early 1980s, but by a different group. With both algorithms enabled, > applications that do two successive writes to a TCP connection, followed > by a read that will not be fulfilled until after the data from the second > write has reached the destination, experience a constant delay of up to > 500 milliseconds, the "ACK delay". It is recommended to disable either, > although traditionally it's easier to disable Nagle, since such a switch > already exists for real-time applications. There is no option to disable delayed acks when creating a socket from Java (or from C either, to the best of my knowledge). So clearly we have to disable Nagle by setting TCP_NODELAY. In order to figure out why you are getting so many small writes to the socket, it would help to know a bit more about your configuration. You are using SSL, right? I think if you used plaintext, sendfile would avoid this issue. What kind of disks are you using? We may find that increasing buffering in the SSL layer would avoid this issue without the drawbacks of enabling Nagle's algorithm. best, Colin On Sun, Mar 21, 2021, at 19:51, Chia-Ping Tsai wrote: > hi Andrei > > Thanks for this find and KIP. Two questions is left. > > 1) Is there significant side effect if we disable it? For example, does > it cause higher latency in your cluster? If there is no obvious side > effect, could we disable it by default? > > 2) As you mentioned that most packets belong to inter broker messages, > is it worth enabling user to use different value (of this new config) > to different listener? For example, users can disable it on controller > listener and enable it on data listener? > > > On 2021/03/17 15:32:47 Andrei Iatsuk wrote: > > Hello everyone, > > > > I would like to start a discussion on KIP-723, which propose adding a Kafka > > Config property with TCP_NODELAY socket option flag, that currently > > hardcoded in true. > > > > https://cwiki.apache.org/confluence/display/KAFKA/KIP-723%3A+Add+socket.tcp.no.delay+property+to+Kafka+Config > > > > Best regards, > > Andrei Iatsuk >
