Hi,

I am using kafka-0.8.0-src.

Cheers,

Klaus


On Fri, Jan 10, 2014 at 8:48 PM, Neha Narkhede <neha.narkh...@gmail.com>wrote:

> What version of Kafka are you benchmarking?
>
>
> On Fri, Jan 10, 2014 at 8:36 AM, Klaus Schaefers <
> klaus.schaef...@ligatus.com> wrote:
>
> > Hi,
> >
> > during my test cpu load is quite low, roughly 50 percent, sometimes peask
> > to 70%.
> >
> > >Are you using the sync producer per chance?
> > I enforced now the async and I got a huge improvement in one of my test
> > cases. let my explore the rest a little bit more.
> >
> > Cheers,
> >
> > Klaus
> >
> >
> >
> >
> > On Fri, Jan 10, 2014 at 2:36 PM, Magnus Edenhill <mag...@edenhill.se>
> > wrote:
> >
> > > 2k msgs/s is silly, unless your messages are 10MB each, so something is
> > > clearly wrong.
> > >
> > > * What is the CPU usage and IO load when running your performance
> tests?
> > > * Are you using the sync producer per chance? Maybe combined with an
> > > aggressive log.flush.interval?
> > > * For reference, and to find out where the bottleneck is, try running
> > > rdkafka_performance [1] according to the previous link I posted:
> > >   if performance does not increase then the broker is the problem,
> > > otherwise it is the producer/consumers.
> > >
> > > Regards,
> > > Magnus
> > >
> > > [1]: https://github.com/edenhill/librdkafka/tree/master/examples
> > >
> > >
> > >
> > > 2014/1/10 Klaus Schaefers <klaus.schaef...@ligatus.com>
> > >
> > > > Hi,
> > > >
> > > > I have close to 2k messages per second. My machine is just a (BG
> 4-core
> > > i5
> > > > but I would expect more messages. I ran Kafka in the default
> settings.
> > > >
> > > >
> > > > On Fri, Jan 10, 2014 at 12:31 PM, Magnus Edenhill <
> mag...@edenhill.se
> > > > >wrote:
> > > >
> > > > > What performance numbers did you see?
> > > > >
> > > > > For reference you can check the following tests that were also run
> on
> > > the
> > > > > same machine as the broker:
> > > > >
> > > > >
> > > >
> > >
> >
> https://github.com/edenhill/librdkafka/blob/master/INTRODUCTION.md#performance-numbers
> > > > >
> > > > > Do they correspond to your numbers?
> > > > >
> > > > > Consumer thruput is not included in that document but typically
> peaks
> > > at
> > > > > around 3 million msgs/s when run on the same host and with hot disk
> > > > caches.
> > > > >
> > > > > Regards,
> > > > > Magnus
> > > > >
> > > > >
> > > > > 2014/1/10 Klaus Schaefers <klaus.schaef...@ligatus.com>
> > > > >
> > > > > > Hi,
> > > > > >
> > > > > > I am currently benchmarking Kafka against ActiveMQ and I got some
> > > > results
> > > > > > the surprised my quite a lot. ActiveMQ managed to deliver 4x more
> > > > > messages
> > > > > > when running locally. But from all what I was reading I am a
> little
> > > bit
> > > > > > surprised. Honestly I expected Kafka to outperform ActiveMQ.
> Some I
> > > am
> > > > > left
> > > > > > with the feeling that I configured some wrong or did any kind of
> > > other
> > > > > > mistake.
> > > > > >
> > > > > > My setup looks like this:
> > > > > >
> > > > > > - One local broker
> > > > > > - 10 Topic / Queues
> > > > > > - 1 producer that dispatches messages randomly to the topics
> > > > > > - 1 consumer per topic
> > > > > >
> > > > > > I basically used the example on the Kafka web page.
> > > > > >
> > > > > > Also I encountered some issues when increasing the number of
> > topics,
> > > > lets
> > > > > > say 100. In this case the consumer cannot connect the
> Zookeeper...
> > > > > >
> > > > > > Does anybody has an idea how to improve the performance?
> > > > > >
> > > > > >
> > > > > > Thx,
> > > > > >
> > > > > > Klaus
> > > > > >
> > > > > >
> > > > > > --
> > > > > >
> > > > > > --
> > > > > >
> > > > > > Klaus Schaefers
> > > > > > Senior Optimization Manager
> > > > > >
> > > > > > Ligatus GmbH
> > > > > > Hohenstaufenring 30-32
> > > > > > D-50674 Köln
> > > > > >
> > > > > > Tel.:  +49 (0) 221 / 56939 -784
> > > > > > Fax:  +49 (0) 221 / 56 939 - 599
> > > > > > E-Mail: klaus.schaef...@ligatus.com
> > > > > > Web: www.ligatus.de
> > > > > >
> > > > > > HRB Köln 56003
> > > > > > Geschäftsführung:
> > > > > > Dipl.-Kaufmann Lars Hasselbach, Dipl.-Kaufmann Klaus Ludemann,
> > > > > > Dipl.-Wirtschaftsingenieur Arne Wolter
> > > > > >
> > > > >
> > > >
> > > >
> > > >
> > > > --
> > > >
> > > > --
> > > >
> > > > Klaus Schaefers
> > > > Senior Optimization Manager
> > > >
> > > > Ligatus GmbH
> > > > Hohenstaufenring 30-32
> > > > D-50674 Köln
> > > >
> > > > Tel.:  +49 (0) 221 / 56939 -784
> > > > Fax:  +49 (0) 221 / 56 939 - 599
> > > > E-Mail: klaus.schaef...@ligatus.com
> > > > Web: www.ligatus.de
> > > >
> > > > HRB Köln 56003
> > > > Geschäftsführung:
> > > > Dipl.-Kaufmann Lars Hasselbach, Dipl.-Kaufmann Klaus Ludemann,
> > > > Dipl.-Wirtschaftsingenieur Arne Wolter
> > > >
> > >
> >
> >
> >
> > --
> >
> > --
> >
> > Klaus Schaefers
> > Senior Optimization Manager
> >
> > Ligatus GmbH
> > Hohenstaufenring 30-32
> > D-50674 Köln
> >
> > Tel.:  +49 (0) 221 / 56939 -784
> > Fax:  +49 (0) 221 / 56 939 - 599
> > E-Mail: klaus.schaef...@ligatus.com
> > Web: www.ligatus.de
> >
> > HRB Köln 56003
> > Geschäftsführung:
> > Dipl.-Kaufmann Lars Hasselbach, Dipl.-Kaufmann Klaus Ludemann,
> > Dipl.-Wirtschaftsingenieur Arne Wolter
> >
>



-- 

-- 

Klaus Schaefers
Senior Optimization Manager

Ligatus GmbH
Hohenstaufenring 30-32
D-50674 Köln

Tel.:  +49 (0) 221 / 56939 -784
Fax:  +49 (0) 221 / 56 939 - 599
E-Mail: klaus.schaef...@ligatus.com
Web: www.ligatus.de

HRB Köln 56003
Geschäftsführung:
Dipl.-Kaufmann Lars Hasselbach, Dipl.-Kaufmann Klaus Ludemann,
Dipl.-Wirtschaftsingenieur Arne Wolter

Reply via email to