Hi Apurva, When I try producing single messages to a similarly-configured topic (30 partitions, 3x replication, acks=all), and enable full protocol debugging, I am seeing single-digit ms round-trip-times for the request-response cycle at a protocol level. But, I see other overheads which might be client-side which add up to around 900 - 1500 ms.
I'm using the confluent-kafka-go Go bindings for the librdkafka C client library. There were some buffering settings I could tune to get Produce calls down for the Producer, but the same tunings didn't seem to affect Commit calls of the Consumer. So this looks like it may be a Go or C lib issue, and I'll try taking my findings to the developers of those. Thanks, Gwilym On 14 December 2016 at 18:55, Apurva Mehta <apu...@confluent.io> wrote: > Hi Gwilym, > > What is the latency for synchronously producing to this cluster? Is it also > 1000 to 2000ms? > > Thanks, > Apurva > > On Wed, Dec 14, 2016 at 2:17 AM, Gwilym Evans < > gwilym.ev...@bigcommerce.com> > wrote: > > > Hi folks, > > > > New to the list and new to operating Kafka. I'm trying to find out what a > > reasonable turnaround time for committing offsets is. > > > > I'm running a 0.10.1.0 cluster of 17 brokers plus 3 dedicated zookeeper > > nodes, though the cluster has been upgraded from its starting point at > > 0.10.0.0. The offsets topic is at its default configuration. The > partition > > I'm consuming and committing offsets for has 150 partitions. > > > > While I can steadily consume around 90,000 messages/sec using 1 consumer > > without (synchronously) committing offsets, if I want to do so > > synchronously it takes between 1000ms and 2000ms to complete the commit. > If > > I use multiple consumers to reduce the partition count each is assigned > > (I've tested up to 10) then time-to-commit does _not_ change. > > > > Is this reasonable? Or should it be quicker than this? > > > > For now, my use case is simple topic-splitting: read messages, apply > basic > > filters, discard or resend messages to one or more topics. But I don't > want > > to advance the offsets until I am certain the messages have been written > > back... hence a cycle of 1) read a batch, 2) process, 3) write, 4) > commit. > > > > If more information is needed, I'm happy to supply what I can. > > > > Thanks, > > Gwilym > > >
