Looking at the output of list topics, here is what I think happened. When topic1 and topic3 were created, only brokers 1&2 were online and alive. When topic2 was created, almost all brokers were online. Only brokers that are alive at the time of topic creation can be assigned replicas for the topic. I would suggest ensuring that all brokers are alive and repeating the experiment with 0.8.1.1, which is the latest stable release.
Thanks, Neha On Tue, Jun 24, 2014 at 4:44 PM, Virendra Pratap Singh < vpsi...@yahoo-inc.com.invalid> wrote: > Have a kafka cluster with 10 brokers (kafka 0.8.0). All of the brokers > were setup upfront. None was added later. Default number of partition is > set to 4 and default replication to 2. > Have 3 topics in the system. None of these topics are manually created > upfront, when the cluster is setup. So relying on kafka to automatically > create these topics when the producer(s) send data first time for each of > these topics. > We have multiple producer which will emit data for all of these topics at > any point of time. What it means is that kafka will be hit with producer > request simultaneously from multiple producer for producer request for > these 3 topics. > > What is observed is the topics partitions do not get spread out evenly in > this scenario. There are 10 brokers (ids 1-10) so expectation is that 3 * 4 > = 12 topic partitions should be spread out on all 10 servers. However in > this case the first 2 brokers share most of the load and few partitions are > spread out. The same is true for the replicated instances also. > > Here is the dump of list topic > > topic: topic1 partition: 0 leader: 1 replicas: 1,2 isr: 1,2 > topic: topic1 partition: 1 leader: 2 replicas: 2,1 isr: 2,1 > topic: topic1 partition: 2 leader: 1 replicas: 1,2 isr: 1,2 > topic: topic1 partition: 3 leader: 2 replicas: 2,1 isr: 2,1 > topic: topic2 partition: 0 leader: 9 replicas: 9,4 isr: > 9,4 > topic: topic2 partition: 1 leader: 10 replicas: 10,5 isr: > 10,5 > topic: topic2 partition: 2 leader: 1 replicas: 1,6 isr: > 1,6 > topic: topic2 partition: 3 leader: 2 replicas: 2,7 isr: > 2,7 > topic: topic3 partition: 0 leader: 2 replicas: 2,1 isr: 2,1 > topic: topic3 partition: 1 leader: 1 replicas: 1,2 isr: 1,2 > topic: topic3 partition: 2 leader: 2 replicas: 2,1 isr: 2,1 > topic: topic3 partition: 3 leader: 1 replicas: 1,2 isr: 1,2 > > So what is my options to have kafka evenly distribute the topic > partitions? Would pre creating the topics via create topic command help? > > Regards, > Virendra >
