Gary, I was asking last week on the dev list regarding performance in 0.9.x and how best to achieve optimal message throughput and find your results rather interesting.
Is producing 7142 msg/sec a fairly typical rate for your test environment (I realize you're just using your laptop, though). Are you able to share your peak message rates in your target/production environment under peak load? Also--have you noticed any increase/decrease moving from 0.8.x to 0.9.x (if applicable)? I'm attempting to compare producing and consuming rates using Kafka (0.8.x and 0.9.x) and our own in-house low overhead library and would be interested in learning how others are faring. ________________________________________ From: Gary Gershon [g...@intermediasciences.com] Sent: Sunday, December 13, 2015 5:22 PM To: users@kafka.apache.org Subject: Re: Kafka Producer 0.9 performance issue with small messages Guozhang, Yes - you identified the problem! We had inserted the .get() for debugging, but didn’t think of the (huge!) side-effects. Using the async callback works perfectly well. We are now able to send 100,000 records in 14 sec from a laptop to the Bluemix cloud - ~1000x faster, Thank you very much! Gary > On Dec 13, 2015, at 2:48 PM, Guozhang Wang <wangg...@gmail.com> wrote: > > Gary, > > You are calling "kafkaProducer.send(record).get();" for each message, the > get() call block until the Future is initialized, which effectively > synchronize all message sent by asking for the ACK for each message before > sending the next message, hence no batching. > > You can try using "send(record, callback)" for async sending and let the > callback handle errors from the returned metadata. > > Guozhang > > <snipped>
