Re: Kafka Producer 0.9 performance issue with small messages

Dave, This is a new app being developed to use the Bluemix Messaging Hub 0.9 beta service. No migration involved from 0.8. With batching working for us (having fixed our own coding bug), we’re quite pleased with the numbers we’re seeing. Our mindset is that in using a cloud service solution,

RE: Kafka Producer 0.9 performance issue with small messages

our own in-house low overhead library and would be interested in learning how others are faring. From: Gary Gershon [g...@intermediasciences.com] Sent: Sunday, December 13, 2015 5:22 PM To: users@kafka.apache.org Subject: Re: Kafka Producer 0.9 performance

Re: Kafka Producer 0.9 performance issue with small messages

Guozhang, Yes - you identified the problem! We had inserted the .get() for debugging, but didn’t think of the (huge!) side-effects. Using the async callback works perfectly well. We are now able to send 100,000 records in 14 sec from a laptop to the Bluemix cloud - ~1000x faster, Thank you v

Re: Kafka Producer 0.9 performance issue with small messages

Gary, You are calling "kafkaProducer.send(record).get();" for each message, the get() call block until the Future is initialized, which effectively synchronize all message sent by asking for the ACK for each message before sending the next message, hence no batching. You can try using "send(recor

Re: Kafka Producer 0.9 performance issue with small messages

Guozhang, Correct. Even with linger.ms at 1000 ms, the metrics indicate no batching. The KafkaProducer instance is recognizing the linger.ms setting since sending 100 records with linger.ms=1000 then takes over 100 seconds. Here is the test harness. There is an abstract parent class with met

Re: Kafka Producer 0.9 performance issue with small messages

Gary, So you observed records-per-request-avg = 1.0 still with linger.ms = 100 or 1000? It seems you are not use Kafka's ProducerPerformance but are using your own ExploreProducer implementation, could you elaborate a bit about how messages are piped to this client? Guozhang On Sat, Dec 12, 20

Kafka Producer 0.9 performance issue with small messages

I’m moving this issue from Stack Overflow to the Apache Kafka Users List http://stackoverflow.com/questions/34213272/kafka-producer-0-9-performance-issue-with-small-messages There was some dis