Hi, during my test cpu load is quite low, roughly 50 percent, sometimes peask to 70%.
>Are you using the sync producer per chance? I enforced now the async and I got a huge improvement in one of my test cases. let my explore the rest a little bit more. Cheers, Klaus On Fri, Jan 10, 2014 at 2:36 PM, Magnus Edenhill <mag...@edenhill.se> wrote: > 2k msgs/s is silly, unless your messages are 10MB each, so something is > clearly wrong. > > * What is the CPU usage and IO load when running your performance tests? > * Are you using the sync producer per chance? Maybe combined with an > aggressive log.flush.interval? > * For reference, and to find out where the bottleneck is, try running > rdkafka_performance [1] according to the previous link I posted: > if performance does not increase then the broker is the problem, > otherwise it is the producer/consumers. > > Regards, > Magnus > > [1]: https://github.com/edenhill/librdkafka/tree/master/examples > > > > 2014/1/10 Klaus Schaefers <klaus.schaef...@ligatus.com> > > > Hi, > > > > I have close to 2k messages per second. My machine is just a (BG 4-core > i5 > > but I would expect more messages. I ran Kafka in the default settings. > > > > > > On Fri, Jan 10, 2014 at 12:31 PM, Magnus Edenhill <mag...@edenhill.se > > >wrote: > > > > > What performance numbers did you see? > > > > > > For reference you can check the following tests that were also run on > the > > > same machine as the broker: > > > > > > > > > https://github.com/edenhill/librdkafka/blob/master/INTRODUCTION.md#performance-numbers > > > > > > Do they correspond to your numbers? > > > > > > Consumer thruput is not included in that document but typically peaks > at > > > around 3 million msgs/s when run on the same host and with hot disk > > caches. > > > > > > Regards, > > > Magnus > > > > > > > > > 2014/1/10 Klaus Schaefers <klaus.schaef...@ligatus.com> > > > > > > > Hi, > > > > > > > > I am currently benchmarking Kafka against ActiveMQ and I got some > > results > > > > the surprised my quite a lot. ActiveMQ managed to deliver 4x more > > > messages > > > > when running locally. But from all what I was reading I am a little > bit > > > > surprised. Honestly I expected Kafka to outperform ActiveMQ. Some I > am > > > left > > > > with the feeling that I configured some wrong or did any kind of > other > > > > mistake. > > > > > > > > My setup looks like this: > > > > > > > > - One local broker > > > > - 10 Topic / Queues > > > > - 1 producer that dispatches messages randomly to the topics > > > > - 1 consumer per topic > > > > > > > > I basically used the example on the Kafka web page. > > > > > > > > Also I encountered some issues when increasing the number of topics, > > lets > > > > say 100. In this case the consumer cannot connect the Zookeeper... > > > > > > > > Does anybody has an idea how to improve the performance? > > > > > > > > > > > > Thx, > > > > > > > > Klaus > > > > > > > > > > > > -- > > > > > > > > -- > > > > > > > > Klaus Schaefers > > > > Senior Optimization Manager > > > > > > > > Ligatus GmbH > > > > Hohenstaufenring 30-32 > > > > D-50674 Köln > > > > > > > > Tel.: +49 (0) 221 / 56939 -784 > > > > Fax: +49 (0) 221 / 56 939 - 599 > > > > E-Mail: klaus.schaef...@ligatus.com > > > > Web: www.ligatus.de > > > > > > > > HRB Köln 56003 > > > > Geschäftsführung: > > > > Dipl.-Kaufmann Lars Hasselbach, Dipl.-Kaufmann Klaus Ludemann, > > > > Dipl.-Wirtschaftsingenieur Arne Wolter > > > > > > > > > > > > > > > -- > > > > -- > > > > Klaus Schaefers > > Senior Optimization Manager > > > > Ligatus GmbH > > Hohenstaufenring 30-32 > > D-50674 Köln > > > > Tel.: +49 (0) 221 / 56939 -784 > > Fax: +49 (0) 221 / 56 939 - 599 > > E-Mail: klaus.schaef...@ligatus.com > > Web: www.ligatus.de > > > > HRB Köln 56003 > > Geschäftsführung: > > Dipl.-Kaufmann Lars Hasselbach, Dipl.-Kaufmann Klaus Ludemann, > > Dipl.-Wirtschaftsingenieur Arne Wolter > > > -- -- Klaus Schaefers Senior Optimization Manager Ligatus GmbH Hohenstaufenring 30-32 D-50674 Köln Tel.: +49 (0) 221 / 56939 -784 Fax: +49 (0) 221 / 56 939 - 599 E-Mail: klaus.schaef...@ligatus.com Web: www.ligatus.de HRB Köln 56003 Geschäftsführung: Dipl.-Kaufmann Lars Hasselbach, Dipl.-Kaufmann Klaus Ludemann, Dipl.-Wirtschaftsingenieur Arne Wolter
