Hi Gwilym, What is the latency for synchronously producing to this cluster? Is it also 1000 to 2000ms?
Thanks, Apurva On Wed, Dec 14, 2016 at 2:17 AM, Gwilym Evans <gwilym.ev...@bigcommerce.com> wrote: > Hi folks, > > New to the list and new to operating Kafka. I'm trying to find out what a > reasonable turnaround time for committing offsets is. > > I'm running a 0.10.1.0 cluster of 17 brokers plus 3 dedicated zookeeper > nodes, though the cluster has been upgraded from its starting point at > 0.10.0.0. The offsets topic is at its default configuration. The partition > I'm consuming and committing offsets for has 150 partitions. > > While I can steadily consume around 90,000 messages/sec using 1 consumer > without (synchronously) committing offsets, if I want to do so > synchronously it takes between 1000ms and 2000ms to complete the commit. If > I use multiple consumers to reduce the partition count each is assigned > (I've tested up to 10) then time-to-commit does _not_ change. > > Is this reasonable? Or should it be quicker than this? > > For now, my use case is simple topic-splitting: read messages, apply basic > filters, discard or resend messages to one or more topics. But I don't want > to advance the offsets until I am certain the messages have been written > back... hence a cycle of 1) read a batch, 2) process, 3) write, 4) commit. > > If more information is needed, I'm happy to supply what I can. > > Thanks, > Gwilym >
