Gary,

I was asking last week on the dev list regarding performance in 0.9.x and how 
best to achieve optimal message throughput and find your results rather 
interesting.

Is producing 7142 msg/sec a fairly typical rate for your test environment (I 
realize you're  just using your laptop, though).  Are you able to share your 
peak message rates in your target/production environment under peak load?

Also--have you noticed any increase/decrease moving from 0.8.x to 0.9.x (if 
applicable)?

I'm attempting to compare producing and consuming rates using Kafka (0.8.x and 
0.9.x) and our own in-house low overhead library and would be interested in 
learning how others are faring.

________________________________________
From: Gary Gershon [g...@intermediasciences.com]
Sent: Sunday, December 13, 2015 5:22 PM
To: users@kafka.apache.org
Subject: Re: Kafka Producer 0.9 performance issue with small messages

Guozhang,

Yes - you identified the problem!

We had inserted the .get() for debugging, but didn’t think of the (huge!) 
side-effects.

Using the async callback works perfectly well.

We are now able to send 100,000 records in 14 sec from a laptop to the Bluemix 
cloud - ~1000x faster,

Thank you very much!

Gary


> On Dec 13, 2015, at 2:48 PM, Guozhang Wang <wangg...@gmail.com> wrote:
>
> Gary,
>
> You are calling "kafkaProducer.send(record).get();" for each message, the
> get() call block until the Future is initialized, which effectively
> synchronize all message sent by asking for the ACK for each message before
> sending the next message, hence no batching.
>
> You can try using "send(record, callback)" for async sending and let the
> callback handle errors from the returned metadata.
>
> Guozhang
>
> <snipped>

Reply via email to