These numbers are a bit misleading. In Kafka, a topic partition is the smallest unit that we distribute messages among consumers in the same consumer group. So, if the number of consumers is larger than the total number of partitions in a Kafka cluster, some consumers will never get any data.
In your experiment with 2 partitions, only 2 of the consumers will get data. The other 2 will not get any data. The 2 consumers that get data will have a higher message rate. However, the aggregate consumption throughput doesn't change (compared with the case with 10 partitions). Thanks, Jun On Wed, Dec 11, 2013 at 12:38 AM, Arjun <ar...@socialtwist.com> wrote: > Hi, > > I am trying my hands on kafka 0.8. I have 3 kafka servers and 3 zookeepers > running.With the number of partitions as 10 and replication factor of 2, 4 > producers were pushing data into kafka, each has their own topic. There are > 4 consumers which are getting the data from kafka. > > The problem i see is with the the number of partitions as 10, consumers > are not consuming enough messages, the through put is almost all half, as > supposed to when the number of partitions is 2. > > Is this a normal scenario or can we tune this behaviour so that the > throughput will be high, when the number of partitions are high. > > environment 1 : > kafka servers : 3 > Zookeepers : 3 > > producers : 4 > consumers : 4 > topics : 4 > > number of partitions: 10 > replication factor : 2 > > number of messages pushed to kafka in 10 minutes : 47935 (per > producer, all producers have pushed almost same number of messages) > number of messages consumed from kafka in same 10 minutes : 20693 (per > consumer, all consumers have consumed almost same number of messages) > > environment 2 : > kafka servers : 3 > Zookeepers : 3 > > producers : 4 > consumers : 4 > topics : 4 > > number of partitions: *2* > replication factor : 2 > > number of messages pushed to kafka in 10 minutes : 47935 (per > producer, all producers have pushed almost same number of messages) > number of messages consumed from kafka in same 10 minutes : 47720 (per > consumer, all consumers have consumed almost same number of messages) > > > Thanks > Arjun Narasimha Kota >
