What version of Kafka are you benchmarking?
On Fri, Jan 10, 2014 at 8:36 AM, Klaus Schaefers < klaus.schaef...@ligatus.com> wrote: > Hi, > > during my test cpu load is quite low, roughly 50 percent, sometimes peask > to 70%. > > >Are you using the sync producer per chance? > I enforced now the async and I got a huge improvement in one of my test > cases. let my explore the rest a little bit more. > > Cheers, > > Klaus > > > > > On Fri, Jan 10, 2014 at 2:36 PM, Magnus Edenhill <mag...@edenhill.se> > wrote: > > > 2k msgs/s is silly, unless your messages are 10MB each, so something is > > clearly wrong. > > > > * What is the CPU usage and IO load when running your performance tests? > > * Are you using the sync producer per chance? Maybe combined with an > > aggressive log.flush.interval? > > * For reference, and to find out where the bottleneck is, try running > > rdkafka_performance [1] according to the previous link I posted: > > if performance does not increase then the broker is the problem, > > otherwise it is the producer/consumers. > > > > Regards, > > Magnus > > > > [1]: https://github.com/edenhill/librdkafka/tree/master/examples > > > > > > > > 2014/1/10 Klaus Schaefers <klaus.schaef...@ligatus.com> > > > > > Hi, > > > > > > I have close to 2k messages per second. My machine is just a (BG 4-core > > i5 > > > but I would expect more messages. I ran Kafka in the default settings. > > > > > > > > > On Fri, Jan 10, 2014 at 12:31 PM, Magnus Edenhill <mag...@edenhill.se > > > >wrote: > > > > > > > What performance numbers did you see? > > > > > > > > For reference you can check the following tests that were also run on > > the > > > > same machine as the broker: > > > > > > > > > > > > > > https://github.com/edenhill/librdkafka/blob/master/INTRODUCTION.md#performance-numbers > > > > > > > > Do they correspond to your numbers? > > > > > > > > Consumer thruput is not included in that document but typically peaks > > at > > > > around 3 million msgs/s when run on the same host and with hot disk > > > caches. > > > > > > > > Regards, > > > > Magnus > > > > > > > > > > > > 2014/1/10 Klaus Schaefers <klaus.schaef...@ligatus.com> > > > > > > > > > Hi, > > > > > > > > > > I am currently benchmarking Kafka against ActiveMQ and I got some > > > results > > > > > the surprised my quite a lot. ActiveMQ managed to deliver 4x more > > > > messages > > > > > when running locally. But from all what I was reading I am a little > > bit > > > > > surprised. Honestly I expected Kafka to outperform ActiveMQ. Some I > > am > > > > left > > > > > with the feeling that I configured some wrong or did any kind of > > other > > > > > mistake. > > > > > > > > > > My setup looks like this: > > > > > > > > > > - One local broker > > > > > - 10 Topic / Queues > > > > > - 1 producer that dispatches messages randomly to the topics > > > > > - 1 consumer per topic > > > > > > > > > > I basically used the example on the Kafka web page. > > > > > > > > > > Also I encountered some issues when increasing the number of > topics, > > > lets > > > > > say 100. In this case the consumer cannot connect the Zookeeper... > > > > > > > > > > Does anybody has an idea how to improve the performance? > > > > > > > > > > > > > > > Thx, > > > > > > > > > > Klaus > > > > > > > > > > > > > > > -- > > > > > > > > > > -- > > > > > > > > > > Klaus Schaefers > > > > > Senior Optimization Manager > > > > > > > > > > Ligatus GmbH > > > > > Hohenstaufenring 30-32 > > > > > D-50674 Köln > > > > > > > > > > Tel.: +49 (0) 221 / 56939 -784 > > > > > Fax: +49 (0) 221 / 56 939 - 599 > > > > > E-Mail: klaus.schaef...@ligatus.com > > > > > Web: www.ligatus.de > > > > > > > > > > HRB Köln 56003 > > > > > Geschäftsführung: > > > > > Dipl.-Kaufmann Lars Hasselbach, Dipl.-Kaufmann Klaus Ludemann, > > > > > Dipl.-Wirtschaftsingenieur Arne Wolter > > > > > > > > > > > > > > > > > > > > > -- > > > > > > -- > > > > > > Klaus Schaefers > > > Senior Optimization Manager > > > > > > Ligatus GmbH > > > Hohenstaufenring 30-32 > > > D-50674 Köln > > > > > > Tel.: +49 (0) 221 / 56939 -784 > > > Fax: +49 (0) 221 / 56 939 - 599 > > > E-Mail: klaus.schaef...@ligatus.com > > > Web: www.ligatus.de > > > > > > HRB Köln 56003 > > > Geschäftsführung: > > > Dipl.-Kaufmann Lars Hasselbach, Dipl.-Kaufmann Klaus Ludemann, > > > Dipl.-Wirtschaftsingenieur Arne Wolter > > > > > > > > > -- > > -- > > Klaus Schaefers > Senior Optimization Manager > > Ligatus GmbH > Hohenstaufenring 30-32 > D-50674 Köln > > Tel.: +49 (0) 221 / 56939 -784 > Fax: +49 (0) 221 / 56 939 - 599 > E-Mail: klaus.schaef...@ligatus.com > Web: www.ligatus.de > > HRB Köln 56003 > Geschäftsführung: > Dipl.-Kaufmann Lars Hasselbach, Dipl.-Kaufmann Klaus Ludemann, > Dipl.-Wirtschaftsingenieur Arne Wolter >
