Interesting, Do you have any way to look into the amount and size of the actual traffic being send. I assume because there are more partitions the actual sending of the messages is separated into more messages, leading to more overhead.
On Sat, Feb 20, 2016 at 12:57 PM John Yost <hokiege...@gmail.com> wrote: > Hi Everyone, > > I discovered yesterday completely by accident. When I went from writing to > 10 topics each with 10 partitions to 10 topics each with 2 partitions--all > other config params are the same--my write throughput almost doubled! I > was not expecting this as I've always thought--and, admittedly, I am still > coming up to speed on Kafka--that increasing parallelism via increasing the > number of partitions and/or topics would increase write throughput. I was > actually experimenting going from 10 partitions per topic to 20 partitions > in an effort to increase throughput but stumbled upon 10X2. > > Important notes: > > 1. Replication factor is 1 > 2. async producer > 3. request.required.acks is 1 > > Any ideas? > > --John >
