Thanks for sharing this, Jason. I had performed similar benchmarks back in 2011 (Results are on slide 27 here - https://cwiki.apache.org/confluence/download/attachments/27822226/F_1330_Narkhede_Kafka+%281%29.pptx?version=1&modificationDate=1358352559000). My tests saturated the 1Gb network link between the producer and broker. I'm guessing like you say, it might be able to go higher with more brokers.
I'm not sure how those values will hold up for 0.8, at least in the initial release. Would love to see your performance analysis in a short blog post :) Thanks, Neha On Thu, May 23, 2013 at 1:24 PM, Oleg Ruchovets <oruchov...@gmail.com>wrote: > Can you please share Kafka version you've used for the tests? > > > On Thu, May 23, 2013 at 8:57 PM, Jason Weiss <jason_we...@rapid7.com> > wrote: > > > Folks, > > > > As I posted to the group here yesterday, my 3 server test in AWS produced > > an average of 273,132 events per second with a fixed-size 2K message > > payload. (Please see that thread for details.) In order to determine the > > horizontal scalability, I added an additional server and MORE producer > > clients into the mix. With 3 servers in the cluster, the average was > 91,044 > > eps per server (273,132). After adding a 4th server to the mix, and an > > additional set of producers, I achieved an extended run of 361,320 events > > per second with the same fixed-size 2K message payload. The new server > > added an additional 88,188 events per second throughput capacity to the > > cluster – near perfect horizontal scaling. > > > > While there may be a limit, achieving 500,00 or 750,000 or even 1 million > > events per second with a Kafka cluster isn't beyond reach for the truly > > motivated. > > > > Nice work committers, very nice work. > > > > Jason > > > > > > > > This electronic message contains information which may be confidential or > > privileged. The information is intended for the use of the individual or > > entity named above. If you are not the intended recipient, be aware that > > any disclosure, copying, distribution or use of the contents of this > > information is prohibited. If you have received this electronic > > transmission in error, please notify us by e-mail at ( > > postmas...@rapid7.com) immediately. > > >
