Hello everyone, I'm interested in the maximum transfer rate of a Kafka broker. There is a fair bit of performance figures in terms of messages per second, but I'm more interested in MB/s with rather large messages, let's say 500 kB as an example.
I tried testing by writing with the apache kafka performance test tool, and with librdkafka. The hardware can write continously 700 MB/s to local storage, but the maximum I can achieve with Kafka is about 260 MB/s with acks=0. I varied the various server side and client side options, batch sizes, message sizes, send and receive buffer sizes, network buffer sizes, JVM heap size. I tested both over loopback and over 10Gb ethernet, same results. Before digging into this much more, I wondered whether somebody has experience with writing to a single Kafka broker at these transfer rates? Best regards Dominik
