Hi Neha,
You are correct. I checked the controller.log and found that even
though I had assumed that the producers were started after whole kafka
cluster, that was not true.
And topic1 and topic3 creation request came in only when broker 1 and 2
were alive. And then in split second all the other 8 brokers were up,
followed by topic2 creation request (and so for that we see even
distribution).
Regards,
Virendra
On 6/24/14, 11:06 PM, "Neha Narkhede" <neha.narkh...@gmail.com> wrote:
>Looking at the output of list topics, here is what I think happened. When
>topic1 and topic3 were created, only brokers 1&2 were online and alive.
>When topic2 was created, almost all brokers were online. Only brokers that
>are alive at the time of topic creation can be assigned replicas for the
>topic. I would suggest ensuring that all brokers are alive and repeating
>the experiment with 0.8.1.1, which is the latest stable release.
>
>Thanks,
>Neha
>
>
>On Tue, Jun 24, 2014 at 4:44 PM, Virendra Pratap Singh <
>vpsi...@yahoo-inc.com.invalid> wrote:
>
>> Have a kafka cluster with 10 brokers (kafka 0.8.0). All of the brokers
>> were setup upfront. None was added later. Default number of partition is
>> set to 4 and default replication to 2.
>> Have 3 topics in the system. None of these topics are manually created
>> upfront, when the cluster is setup. So relying on kafka to automatically
>> create these topics when the producer(s) send data first time for each
>>of
>> these topics.
>> We have multiple producer which will emit data for all of these topics
>>at
>> any point of time. What it means is that kafka will be hit with producer
>> request simultaneously from multiple producer for producer request for
>> these 3 topics.
>>
>> What is observed is the topics partitions do not get spread out evenly
>>in
>> this scenario. There are 10 brokers (ids 1-10) so expectation is that 3
>>* 4
>> = 12 topic partitions should be spread out on all 10 servers. However in
>> this case the first 2 brokers share most of the load and few partitions
>>are
>> spread out. The same is true for the replicated instances also.
>>
>> Here is the dump of list topic
>>
>> topic: topic1 partition: 0 leader: 1 replicas: 1,2 isr: 1,2
>> topic: topic1 partition: 1 leader: 2 replicas: 2,1 isr: 2,1
>> topic: topic1 partition: 2 leader: 1 replicas: 1,2 isr: 1,2
>> topic: topic1 partition: 3 leader: 2 replicas: 2,1 isr: 2,1
>> topic: topic2 partition: 0 leader: 9 replicas: 9,4
>>isr:
>> 9,4
>> topic: topic2 partition: 1 leader: 10 replicas: 10,5
>>isr:
>> 10,5
>> topic: topic2 partition: 2 leader: 1 replicas: 1,6
>>isr:
>> 1,6
>> topic: topic2 partition: 3 leader: 2 replicas: 2,7
>>isr:
>> 2,7
>> topic: topic3 partition: 0 leader: 2 replicas: 2,1 isr:
>>2,1
>> topic: topic3 partition: 1 leader: 1 replicas: 1,2 isr:
>>1,2
>> topic: topic3 partition: 2 leader: 2 replicas: 2,1 isr:
>>2,1
>> topic: topic3 partition: 3 leader: 1 replicas: 1,2 isr:
>>1,2
>>
>> So what is my options to have kafka evenly distribute the topic
>> partitions? Would pre creating the topics via create topic command help?
>>
>> Regards,
>> Virendra
>>
