The following tests were using a single producer application (examples/rdkafka_performance):
* Test1: 2 brokers, 2 partitions, required.acks=2, 100 byte messages: 850000 messages/second, 85 MB/second * Test2: 1 broker, 1 partition, required.acks=0, 100 byte messages: 710000 messages/second, 71 MB/second * Test3: 2 broker2, 2 partitions, required.acks=2, 100 byte messages, snappy compression: 300000 messages/second, 30 MB/second * Test4: 2 broker2, 2 partitions, required.acks=2, 100 byte messages, gzip compression: 230000 messages/second, 23 MB/second log.flush broker configuration was increased to avoid the disk being the bottleneck. /Magnus 2013/11/24 Neha Narkhede <neha.narkh...@gmail.com> > So, a single producer'a throughput is 80 MB/s? That seems pretty high. What > was the number of acks setting? Thanks for sharing these numbers. > > On Sunday, November 24, 2013, Magnus Edenhill wrote: > > > Hi Neha, > > > > these tests were done using 100 byte messages. More information about the > > producer performance tests can be found here: > > > > > https://github.com/edenhill/librdkafka/blob/master/INTRODUCTION.md#performance-numbers > > > > The tests are indicative at best and in no way scientific, but I must say > > that the Kafka broker performance is impressive. > > > > Regards, > > Magnus > > > > > > > > 2013/11/22 Neha Narkhede <neha.narkh...@gmail.com <javascript:;>> > > > > > Thanks for sharing this! What is the message size for the throughput > > > numbers stated below? > > > > > > Thanks, > > > Neha > > > On Nov 22, 2013 6:59 AM, "Magnus Edenhill" <mag...@edenhill.se > <javascript:;>> > > wrote: > > > > > > > This announces the 0.8.0 release of librdkafka - The Apache Kafka > > client > > > C > > > > library - now with 0.8 protocol support. > > > > > > > > Features: > > > > * Producer (~800K msgs/s) > > > > * Consumer (~3M msgs/s) > > > > * Compression (Snappy, gzip) > > > > * Proper failover and leader re-election support - no message is ever > > > lost. > > > > * Configuration properties compatible with official Apache Kafka. > > > > * Stabilized ABI-safe API > > > > * Mainline Debian package submitted > > > > * Production quality > > > > > > > > > > > > Home: > > > > https://github.com/edenhill/librdkafka > > > > > > > > Introduction and performance numbers: > > > > https://github.com/edenhill/librdkafka/blob/master/INTRODUCTION.md > > > > > > > > Have fun. > > > > > > > > Regards, > > > > Magnus > > > > > > > > P.S. > > > > Check out Wikimedia Foundation's varnishkafka daemon for a use case - > > > > varnish log forwarding over Kafka: > > > > https://github.com/wikimedia/varnishkafka > > > > > > > > > >
