You can try to set "queue.buffering.max.messages" larger, for example:
confParam=["queue.buffering.max.messages=2000000", "batch.num.messages=1000"] And reference for you: https://github.com/edenhill/librdkafka/issues/210 2015-10-27 1:19 GMT+08:00 Prabhjot Bharaj <prabhbha...@gmail.com>: > Hi, > > This is a type of problem where you operating more than network capacity. > This can be handled at two places (you decide whichever is useful/practical > for use case) :- > > 1. In case the bottleneck is because of the broker slowness, increase the > number of partitions of your topic, balance them out to newer nodes (if > required) and retry. Make sure your producer can write to the new > partitions as well > 2. In case this is because your producer is creating too many messages, > there are 2 ways you can commit them (select the most appropriate according > to your use case. I'm considering that you dont want to lose your > messages):- > ------> a. increase the number of producer machines (i.e. balance your > producer load among more machines) > -------> b. produce at a slower rate (in case your producers are not > scalable, you are in a problem. you can solve this problem by sacrificing > latency). Write to your queue in batches and wait for it to get cleared > before you commit the next batch to your local queue. > > Regards, > Prabcs > > > On Mon, Oct 26, 2015 at 10:41 PM, Eduardo Costa Alfaia < > e.costaalf...@unibs.it> wrote: > > > Hi Magnus > > I think this answer > > c) producing messages at a higher rate than the network or broker can > > handle > > How could I manager this? > > > > > > > On 26 Oct 2015, at 17:45, Magnus Edenhill <mag...@edenhill.se> wrote: > > > > > > c) producing messages at a higher rate than the network or broker can > > > handle > > > > > > -- > > Informativa sulla Privacy: http://www.unibs.it/node/8155 > > > > > > -- > --------------------------------------------------------- > "There are only 10 types of people in the world: Those who understand > binary, and those who don't" >
