Hello, Some numbers - There is a single topic with 3 producers, 4 brokers, 80 partitions and replication factor of 2. Average produce rate is 45 MB/sec (cumulative) and max is at around 250 MB/sec (cumulative), the producer goes out of memory during peak times (buffer memory set to 10G, batch size set to 10MB and linger millis set to 2000 ms). The system metrics on kafka broker machines indicates that the max disk write rate reaches around 20 MB/sec per broker machine (which seems not so great). Producers are producing more than that and thus go out of buffer memory many times. What are the parameters that be tuned to increase broker throughput keeping the number of brokers same ? Tried increasing the number of partitions from 40 to 80, server.background.threads to 20 and varying producer buffer size from 1 MB to 30 MB but did not got any significant improvements.
I understand this might be a very general question but any help regarding which parameters to tune or any direction to look into will be really helpful. Thanks,Parag
