What's your server setup for the brokers and consumers? Generally I'd expect something to be exhausted here and that to end up being the bottleneck.
Thanks Tom Crayford Heroku Kafka On Mon, May 23, 2016 at 7:32 PM, Yazeed Alabdulkarim < y.alabdulka...@gmail.com> wrote: > Hi, > I am running simple experiments to evaluate the scalability of Kafka > consumers with respect to the number of partitions. I assign every consumer > to a specific partition. Each consumer polls the records in its assigned > partition and print the first one, then polls again from the offset of the > printed record until all records are printed. Prior to running the test, I > produce 10 Million records evenly among partitions. After running the test, > I measure the time it took for the consumers to print all the records. I > was expecting Kafka to scale as I increase the number of > consumers/partitions. However, the scalability diminishes as I increase the > number of partitions/consumers, beyond certain number. Going from 1,2,4,8 > the scalability is great as the duration of the test is reduced by the > factor increase of the number of partitions/consumers. However, beyond 8 > consumers/partitions, the duration of the test reaches a steady state. I am > monitoring the resources of my server and didn't see any bottleneck. Am I > missing something here? Shouldn't Kafka consumers scale with the number of > partitions? > -- > Best Regards, > Yazeed Alabdulkarim >
