Hi Alex, Great info, thanks! I asked a related question this AM--is a full queue possibly a symptom of back pressure within Kafka?
--John On Thu, Feb 18, 2016 at 12:38 PM, Alex Loddengaard <a...@confluent.io> wrote: > Hi Saurabh, > > This is occurring because the produce message queue is full when a produce > request is made. The size of the queue is configured > via queue.buffering.max.messages. You can experiment with increasing this > (which will require more JVM heap space), or fiddling with > queue.enqueue.timeout.ms to control the blocking behavior when the queue > is > full. Both of these configuration options are explained here: > > https://kafka.apache.org/08/configuration.html > > I didn't quite follow your last paragraph, so I'm not sure if the following > advice is applicable to you or not. You may also experiment with adding > more producers (either on the same or different machines). > > I hope this helps. > > Alex > > On Thu, Feb 18, 2016 at 2:12 AM, Saurabh Kumar <saurabh...@gmail.com> > wrote: > > > Hi, > > > > We have a Kafka server deployment shared between multiple teams and i > have > > created a topic with multiple partitions on it for pushing some JSON > data. > > > > We have multiple such Kafka producers running from different machines > which > > produce/push data to a Kafka topic. A lot of times i see the following > > exception in the logs : "*Event queue is full of unsent messages, could > not > > send event"* > > > > Any idea how to solve this ? We can not synchronise the volume or timing > of > > different Kafka producers across machines and between multiple processes. > > There is a limit on maximum number of concurrent processes (kafka > producer) > > that can run on a mchine but it is only going to increase with time as > we > > scale. What is the right way to solve this problem ? > > > > Thanks, > > Saurabh > > > > > > -- > *Alex Loddengaard | **Solutions Architect | Confluent* > *Download Apache Kafka and Confluent Platform: www.confluent.io/download > <http://www.confluent.io/download>* >
