Hi, I am using kafka-0.8.0-src.
Cheers, Klaus On Fri, Jan 10, 2014 at 8:48 PM, Neha Narkhede <neha.narkh...@gmail.com>wrote: > What version of Kafka are you benchmarking? > > > On Fri, Jan 10, 2014 at 8:36 AM, Klaus Schaefers < > klaus.schaef...@ligatus.com> wrote: > > > Hi, > > > > during my test cpu load is quite low, roughly 50 percent, sometimes peask > > to 70%. > > > > >Are you using the sync producer per chance? > > I enforced now the async and I got a huge improvement in one of my test > > cases. let my explore the rest a little bit more. > > > > Cheers, > > > > Klaus > > > > > > > > > > On Fri, Jan 10, 2014 at 2:36 PM, Magnus Edenhill <mag...@edenhill.se> > > wrote: > > > > > 2k msgs/s is silly, unless your messages are 10MB each, so something is > > > clearly wrong. > > > > > > * What is the CPU usage and IO load when running your performance > tests? > > > * Are you using the sync producer per chance? Maybe combined with an > > > aggressive log.flush.interval? > > > * For reference, and to find out where the bottleneck is, try running > > > rdkafka_performance [1] according to the previous link I posted: > > > if performance does not increase then the broker is the problem, > > > otherwise it is the producer/consumers. > > > > > > Regards, > > > Magnus > > > > > > [1]: https://github.com/edenhill/librdkafka/tree/master/examples > > > > > > > > > > > > 2014/1/10 Klaus Schaefers <klaus.schaef...@ligatus.com> > > > > > > > Hi, > > > > > > > > I have close to 2k messages per second. My machine is just a (BG > 4-core > > > i5 > > > > but I would expect more messages. I ran Kafka in the default > settings. > > > > > > > > > > > > On Fri, Jan 10, 2014 at 12:31 PM, Magnus Edenhill < > mag...@edenhill.se > > > > >wrote: > > > > > > > > > What performance numbers did you see? > > > > > > > > > > For reference you can check the following tests that were also run > on > > > the > > > > > same machine as the broker: > > > > > > > > > > > > > > > > > > > > https://github.com/edenhill/librdkafka/blob/master/INTRODUCTION.md#performance-numbers > > > > > > > > > > Do they correspond to your numbers? > > > > > > > > > > Consumer thruput is not included in that document but typically > peaks > > > at > > > > > around 3 million msgs/s when run on the same host and with hot disk > > > > caches. > > > > > > > > > > Regards, > > > > > Magnus > > > > > > > > > > > > > > > 2014/1/10 Klaus Schaefers <klaus.schaef...@ligatus.com> > > > > > > > > > > > Hi, > > > > > > > > > > > > I am currently benchmarking Kafka against ActiveMQ and I got some > > > > results > > > > > > the surprised my quite a lot. ActiveMQ managed to deliver 4x more > > > > > messages > > > > > > when running locally. But from all what I was reading I am a > little > > > bit > > > > > > surprised. Honestly I expected Kafka to outperform ActiveMQ. > Some I > > > am > > > > > left > > > > > > with the feeling that I configured some wrong or did any kind of > > > other > > > > > > mistake. > > > > > > > > > > > > My setup looks like this: > > > > > > > > > > > > - One local broker > > > > > > - 10 Topic / Queues > > > > > > - 1 producer that dispatches messages randomly to the topics > > > > > > - 1 consumer per topic > > > > > > > > > > > > I basically used the example on the Kafka web page. > > > > > > > > > > > > Also I encountered some issues when increasing the number of > > topics, > > > > lets > > > > > > say 100. In this case the consumer cannot connect the > Zookeeper... > > > > > > > > > > > > Does anybody has an idea how to improve the performance? > > > > > > > > > > > > > > > > > > Thx, > > > > > > > > > > > > Klaus > > > > > > > > > > > > > > > > > > -- > > > > > > > > > > > > -- > > > > > > > > > > > > Klaus Schaefers > > > > > > Senior Optimization Manager > > > > > > > > > > > > Ligatus GmbH > > > > > > Hohenstaufenring 30-32 > > > > > > D-50674 Köln > > > > > > > > > > > > Tel.: +49 (0) 221 / 56939 -784 > > > > > > Fax: +49 (0) 221 / 56 939 - 599 > > > > > > E-Mail: klaus.schaef...@ligatus.com > > > > > > Web: www.ligatus.de > > > > > > > > > > > > HRB Köln 56003 > > > > > > Geschäftsführung: > > > > > > Dipl.-Kaufmann Lars Hasselbach, Dipl.-Kaufmann Klaus Ludemann, > > > > > > Dipl.-Wirtschaftsingenieur Arne Wolter > > > > > > > > > > > > > > > > > > > > > > > > > > > -- > > > > > > > > -- > > > > > > > > Klaus Schaefers > > > > Senior Optimization Manager > > > > > > > > Ligatus GmbH > > > > Hohenstaufenring 30-32 > > > > D-50674 Köln > > > > > > > > Tel.: +49 (0) 221 / 56939 -784 > > > > Fax: +49 (0) 221 / 56 939 - 599 > > > > E-Mail: klaus.schaef...@ligatus.com > > > > Web: www.ligatus.de > > > > > > > > HRB Köln 56003 > > > > Geschäftsführung: > > > > Dipl.-Kaufmann Lars Hasselbach, Dipl.-Kaufmann Klaus Ludemann, > > > > Dipl.-Wirtschaftsingenieur Arne Wolter > > > > > > > > > > > > > > > -- > > > > -- > > > > Klaus Schaefers > > Senior Optimization Manager > > > > Ligatus GmbH > > Hohenstaufenring 30-32 > > D-50674 Köln > > > > Tel.: +49 (0) 221 / 56939 -784 > > Fax: +49 (0) 221 / 56 939 - 599 > > E-Mail: klaus.schaef...@ligatus.com > > Web: www.ligatus.de > > > > HRB Köln 56003 > > Geschäftsführung: > > Dipl.-Kaufmann Lars Hasselbach, Dipl.-Kaufmann Klaus Ludemann, > > Dipl.-Wirtschaftsingenieur Arne Wolter > > > -- -- Klaus Schaefers Senior Optimization Manager Ligatus GmbH Hohenstaufenring 30-32 D-50674 Köln Tel.: +49 (0) 221 / 56939 -784 Fax: +49 (0) 221 / 56 939 - 599 E-Mail: klaus.schaef...@ligatus.com Web: www.ligatus.de HRB Köln 56003 Geschäftsführung: Dipl.-Kaufmann Lars Hasselbach, Dipl.-Kaufmann Klaus Ludemann, Dipl.-Wirtschaftsingenieur Arne Wolter
