2k msgs/s is silly, unless your messages are 10MB each, so something is clearly wrong.
* What is the CPU usage and IO load when running your performance tests? * Are you using the sync producer per chance? Maybe combined with an aggressive log.flush.interval? * For reference, and to find out where the bottleneck is, try running rdkafka_performance [1] according to the previous link I posted: if performance does not increase then the broker is the problem, otherwise it is the producer/consumers. Regards, Magnus [1]: https://github.com/edenhill/librdkafka/tree/master/examples 2014/1/10 Klaus Schaefers <klaus.schaef...@ligatus.com> > Hi, > > I have close to 2k messages per second. My machine is just a (BG 4-core i5 > but I would expect more messages. I ran Kafka in the default settings. > > > On Fri, Jan 10, 2014 at 12:31 PM, Magnus Edenhill <mag...@edenhill.se > >wrote: > > > What performance numbers did you see? > > > > For reference you can check the following tests that were also run on the > > same machine as the broker: > > > > > https://github.com/edenhill/librdkafka/blob/master/INTRODUCTION.md#performance-numbers > > > > Do they correspond to your numbers? > > > > Consumer thruput is not included in that document but typically peaks at > > around 3 million msgs/s when run on the same host and with hot disk > caches. > > > > Regards, > > Magnus > > > > > > 2014/1/10 Klaus Schaefers <klaus.schaef...@ligatus.com> > > > > > Hi, > > > > > > I am currently benchmarking Kafka against ActiveMQ and I got some > results > > > the surprised my quite a lot. ActiveMQ managed to deliver 4x more > > messages > > > when running locally. But from all what I was reading I am a little bit > > > surprised. Honestly I expected Kafka to outperform ActiveMQ. Some I am > > left > > > with the feeling that I configured some wrong or did any kind of other > > > mistake. > > > > > > My setup looks like this: > > > > > > - One local broker > > > - 10 Topic / Queues > > > - 1 producer that dispatches messages randomly to the topics > > > - 1 consumer per topic > > > > > > I basically used the example on the Kafka web page. > > > > > > Also I encountered some issues when increasing the number of topics, > lets > > > say 100. In this case the consumer cannot connect the Zookeeper... > > > > > > Does anybody has an idea how to improve the performance? > > > > > > > > > Thx, > > > > > > Klaus > > > > > > > > > -- > > > > > > -- > > > > > > Klaus Schaefers > > > Senior Optimization Manager > > > > > > Ligatus GmbH > > > Hohenstaufenring 30-32 > > > D-50674 Köln > > > > > > Tel.: +49 (0) 221 / 56939 -784 > > > Fax: +49 (0) 221 / 56 939 - 599 > > > E-Mail: klaus.schaef...@ligatus.com > > > Web: www.ligatus.de > > > > > > HRB Köln 56003 > > > Geschäftsführung: > > > Dipl.-Kaufmann Lars Hasselbach, Dipl.-Kaufmann Klaus Ludemann, > > > Dipl.-Wirtschaftsingenieur Arne Wolter > > > > > > > > > -- > > -- > > Klaus Schaefers > Senior Optimization Manager > > Ligatus GmbH > Hohenstaufenring 30-32 > D-50674 Köln > > Tel.: +49 (0) 221 / 56939 -784 > Fax: +49 (0) 221 / 56 939 - 599 > E-Mail: klaus.schaef...@ligatus.com > Web: www.ligatus.de > > HRB Köln 56003 > Geschäftsführung: > Dipl.-Kaufmann Lars Hasselbach, Dipl.-Kaufmann Klaus Ludemann, > Dipl.-Wirtschaftsingenieur Arne Wolter >
