Hi List

I am running Kafka 3.7.1 on CentOS 7.9.

I have written a Kafka producer program in Python using confluent_kafka (which 
uses librdkafka).

When the program is running on a machine in the same data centre as the Kafka 
cluster, a single producer writing 1,000 byte messages to a single topic 
partition can sustain approx 55 MB/sec of throughput over a ten minute period.

Produer config:

request.required.acks=all
enable.idempotence=true

When I run the program on a machine in a remote data centre with a high 
round-trip time the throughput drops to approx 1.5 MB/sec.

This is expected, and as noted in the docs 
(https://kafka.apache.org/documentation/#datacenters):

"Kafka naturally batches data in both the producer and consumer so it can 
achieve high-throughput even over a high-latency connection. To allow this 
though it may be necessary to increase the TCP socket buffer sizes for the 
producer, consumer, and broker using the socket.send.buffer.bytes and 
socket.receive.buffer.bytes configurations".

Based on the use-case, I have calculated that 25 MB socket buffers to be 
optimal for my use-case and have made the following OS config changes on the 
client machine and brokers:

net.ipv4.tcp_rmem='4096    87380   27262976'
net.ipv4.tcp_wmem='4096    16384   27262976'
net.core.rmem_max=27262976
net.core.wmem_max=27262976
net.core.rmem_default=27262976
net.core.wmem_default=27262976

and confirmed that net.ipv4.tcp_window_scaling=1

I have also set socket.send.buffer.bytes=27262976 and 
socket.receive.buffer.bytes=27262976 on the brokers and 
socket.send.buffer.bytes=27262976 on the produer.

However, this has made no difference in terms of throughput.

I also tried setting socket.send.buffer.bytes=-1 and 
socket.receive.buffer.bytes=-1 on the brokers and socket.send.buffer.bytes=0 
(i.e. use OS defaults) but this also made no difference.

Note that I have turned on "msg" debug on the producer and confirmed that 
batching is occuring, with a typical MessageSet having ~ 9000 messages.

Also note the broker logs show the expected values for socket.send.buffer.bytes 
and socket.receive.buffer.bytes at startup, so i'm confident the brokers have 
picked up the config property changes after I restarted them.

This led me to suspect that Kafka is not honouring my socket buffer sizing 
settings.

When I look at the output of "ss -tmn" on the broker for the socket that my 
producer is using the recieve buffer is not of the expected size:

====
ESTAB      0      148    XX.XXX.XX.XX:6669               XX.X.XX.XX:58870
         skmem:(r0,rb971074,t0,tb46080,f45056,w0,o0,bl0,d438)
====

Per the above output, the recieve buffer is 971074 bytes (skmem:...rb971074).

I have also set log4j.logger.org.apache.kafka.common.network.Selector=DEBUG in 
log4j.properties on the broker in order to check the size of the recieve buffer 
requested. However, I only see DEBUG entries for RaftManager, 
NodeToControllerChannelManager, and ReplicaFetcher but not the socket my 
producer is writing to as far as i can tell.

if anyone has any suggestions on how I can make sure that Kafka uses my desired 
socket buffer sizes, or other ways of verifying whether or not it is using 
them, it would be much appreciated

Thanks

Austin

Reply via email to