In 0.8.2 beta, we have a new java producer that's more performing. You can give it a try.
Thanks, Jun On Thu, Nov 6, 2014 at 10:16 AM, Devendra Tagare < devendra.tag...@pubmatic.com> wrote: > Hi, > > Which version of Kafka are you using? > 0.8.0 > > Is the broker I/O or network saturated? - typical writes are around 45 MB > per broker & there are streaming as well as batch consumers. > > We are already sending across compressed packets & the batch size is 100. > > Also, we tried the sync producer but were not able to enforce the timeouts > on it.Is there anything that needs to be taken care of in this case? > > Regards, > Dev > ________________________________ > From: Devendra Tagare > Sent: Tuesday, November 04, 2014 10:02 AM > To: users@kafka.apache.org > Cc: Devendra Tagare > Subject: Issue with async producer > > Hi, > > We are using an async producer to send data to kafka. > > The load the sender handles is around 250 rps ,the size of a message is > around 25K. > > The configs used in the producer are : > > request.required.acks=0 > producer.type=async > batch.num.messages=10 > topic.metadata.refresh.interval.ms=30000 > queue.buffering.max.ms=300 > queue.enqueue.timeout.ms=50 > > > While the asnc producer works perfectly fine at 150-175 rps,the invoking > method returns under 10 ms.The invoking method takes around 20000ms to > return when the load increases to 250rps. > > On investigation we noticed QueueFullExceptions in the logs. > > On the kafka side the memory utilization was high.Is is because the fsync > to memory is not happening fast enough? > The document says the OS level fsync interval should take care of the > interval at which writes are happening.Should we expediate the write using > > log.flush.interval.messages > and log.flush.interval.ms. > > Also, we tried the sync producer but were not able to enforce the timeouts > on it. > > We have 45 producers & 11 Kafka servers which handle a load of around > 500mn events per day. > > Some of the server side properties we are using: > > log.segment.bytes=536870912 > log.retention.check.interval.ms=60000 > zookeeper.connection.timeout.ms=1000000 > > Regards, > Dev >
