I tried benchmarking kafka producer with acks=1 in 5 node cluster .. Total transfer rate is ~950MB/sec .. Single broker transfer rate is less than 200MB/sec..
Load Generator: I've started 6 instance of http server where it writes to broker ... Using wrk2 http benchmarking tool I was able to send more events to Kafka Cluster for this benchmarking test.. --Senthil On Jul 3, 2017 2:17 PM, "Dominik Werder" <dominik.wer...@gmail.com> wrote: Hello everyone, I'm interested in the maximum transfer rate of a Kafka broker. There is a fair bit of performance figures in terms of messages per second, but I'm more interested in MB/s with rather large messages, let's say 500 kB as an example. I tried testing by writing with the apache kafka performance test tool, and with librdkafka. The hardware can write continously 700 MB/s to local storage, but the maximum I can achieve with Kafka is about 260 MB/s with acks=0. I varied the various server side and client side options, batch sizes, message sizes, send and receive buffer sizes, network buffer sizes, JVM heap size. I tested both over loopback and over 10Gb ethernet, same results. Before digging into this much more, I wondered whether somebody has experience with writing to a single Kafka broker at these transfer rates? Best regards Dominik
