Ok, then your understanding is correct. If the queue is full then the enqueue operation should result in a kafka.common.QueueFullException in 20 ms. Not sure why this is not happening since in the code, we simply call queue.offer() with the timeout. It could be due to contention. Do you have lots of threads sending data to the same producer instance? Also, could you check the queue size JMX to see if it's really full?
Thanks, Jun On Thu, Nov 20, 2014 at 9:56 PM, Devendra Tagare < devendra.tag...@pubmatic.com> wrote: > Hi, > > We are using an async producer. > > The producer properties are: > > producer.type= async > queue.enqueue.timeout.ms=20 > send.buffer.bytes=1024000 > topic.metadata.refresh.interval.ms=30000 > > > Please find more details in the mail thread below. > > > Regards, > Dev > ________________________________ > From: Devendra Tagare > Sent: Thursday, November 20, 2014 6:31 PM > To: users@kafka.apache.org > Cc: Devendra Tagare > Subject: Issues with Kafka async producer's enqueue timeout > > Hi, > > We are using an async producer to send data to a kafka cluster.The event > rate at peak is around 250 events/second of size 25KB each. > > In the producer code base we have added specific debug statements to > capture the time taken to create a producer,create a keyed message with a > byte payload & send the message. > > We have added the below properties to the ProducerConfig > > producer.type= async > queue.enqueue.timeout.ms=20 > send.buffer.bytes=1024000 > topic.metadata.refresh.interval.ms=30000 > > We also checked if these properties are being picked. > > Based on the documentation, producer.send() queues the message on the > async producer's queue. > > So, ideally if the queue is full then the enqueue operation should result > in an kafka.common.QueueFullException in 20 ms. > > The logs indicate that the enqueue operation is taking more than 20ms > (takes around 250ms) without throwing any exceptions. > > Is there any other property that could conflict with > queue.enqueue.timeout.ms which is causing this behavior ? > > Or is it possible that the queue is not full & yet the producer.send() > call is still taking around 200ms under peak load ? > > Also, could you suggest any other alternatives so that we can either > enforce a timeout or throw an exception in-case the async producer is > taking more than a specified amount of time to queue an event. > > > Regards, > Dev >
