few more things you can do: * Increase "batch.size" - this will give you a larger queue and usually better throughput * More producers - very often the bottleneck is not in Kafka at all. Maybe its the producer? or the network? * Increate max.inflight.requests for the producer - it will allow sending more requests concurrently and perhaps increase throughput.
The important bit is: Don't add more brokers if you don't have information that the broker is the bottleneck. Gwen On Mon, Apr 25, 2016 at 12:06 PM, Saurabh Kumar <saurabh...@gmail.com> wrote: > Thanks Alex and Sorry for the delayed response. We never could solv this > problem so am resurrecting the thread. As i understand, from a > client/tenant which is producing messages to a kafka topic, there is not > much that can be controlled. I assume "Event queue is full of unsent > messages" signify that : > 1) We need to expand our cluster by adding more resources/brokers > 2) We need to add a blocking behaviour incase we see that the average > volume of messages is sustainable, and its just the spikes that are causing > problems. > > --Saurabh > > On Thu, Feb 18, 2016 at 11:51 PM, John Yost <hokiege...@gmail.com> wrote: > >> Hi Alex, >> >> Great info, thanks! I asked a related question this AM--is a full queue >> possibly a symptom of back pressure within Kafka? >> >> --John >> >> On Thu, Feb 18, 2016 at 12:38 PM, Alex Loddengaard <a...@confluent.io> >> wrote: >> >> > Hi Saurabh, >> > >> > This is occurring because the produce message queue is full when a >> produce >> > request is made. The size of the queue is configured >> > via queue.buffering.max.messages. You can experiment with increasing this >> > (which will require more JVM heap space), or fiddling with >> > queue.enqueue.timeout.ms to control the blocking behavior when the queue >> > is >> > full. Both of these configuration options are explained here: >> > >> > https://kafka.apache.org/08/configuration.html >> > >> > I didn't quite follow your last paragraph, so I'm not sure if the >> following >> > advice is applicable to you or not. You may also experiment with adding >> > more producers (either on the same or different machines). >> > >> > I hope this helps. >> > >> > Alex >> > >> > On Thu, Feb 18, 2016 at 2:12 AM, Saurabh Kumar <saurabh...@gmail.com> >> > wrote: >> > >> > > Hi, >> > > >> > > We have a Kafka server deployment shared between multiple teams and i >> > have >> > > created a topic with multiple partitions on it for pushing some JSON >> > data. >> > > >> > > We have multiple such Kafka producers running from different machines >> > which >> > > produce/push data to a Kafka topic. A lot of times i see the following >> > > exception in the logs : "*Event queue is full of unsent messages, could >> > not >> > > send event"* >> > > >> > > Any idea how to solve this ? We can not synchronise the volume or >> timing >> > of >> > > different Kafka producers across machines and between multiple >> processes. >> > > There is a limit on maximum number of concurrent processes (kafka >> > producer) >> > > that can run on a mchine but it is only going to increase with time as >> > we >> > > scale. What is the right way to solve this problem ? >> > > >> > > Thanks, >> > > Saurabh >> > > >> > >> > >> > >> > -- >> > *Alex Loddengaard | **Solutions Architect | Confluent* >> > *Download Apache Kafka and Confluent Platform: www.confluent.io/download >> > <http://www.confluent.io/download>* >> > >>
