Hi folks, New to the list and new to operating Kafka. I'm trying to find out what a reasonable turnaround time for committing offsets is.
I'm running a 0.10.1.0 cluster of 17 brokers plus 3 dedicated zookeeper nodes, though the cluster has been upgraded from its starting point at 0.10.0.0. The offsets topic is at its default configuration. The partition I'm consuming and committing offsets for has 150 partitions. While I can steadily consume around 90,000 messages/sec using 1 consumer without (synchronously) committing offsets, if I want to do so synchronously it takes between 1000ms and 2000ms to complete the commit. If I use multiple consumers to reduce the partition count each is assigned (I've tested up to 10) then time-to-commit does _not_ change. Is this reasonable? Or should it be quicker than this? For now, my use case is simple topic-splitting: read messages, apply basic filters, discard or resend messages to one or more topics. But I don't want to advance the offsets until I am certain the messages have been written back... hence a cycle of 1) read a batch, 2) process, 3) write, 4) commit. If more information is needed, I'm happy to supply what I can. Thanks, Gwilym
