Hi Todd, That worked. Basically I used a python library to read the leaders of a topic/partition from the following path on zookeeper: /brokers/topics/[topic]/partitions/[partitionId]/state.
Then I was able to get the correct leader and assign the replica set so that the current leader remains the first entry in that list, basically to keep the preferred replica logic intact. Thanks for the help. Thanks! Sagar. On Fri, Dec 22, 2017 at 10:08 PM, Todd Palino <tpal...@gmail.com> wrote: > Yes, the replicas are stored in Zookeeper, so you can iterate over the > information there to build a view of the cluster that you can use. If you > want an example for this, take a look at the code for kafka-assigner in > https://github.com/linkedin/kafka-tools. Or you can just use that tool to > adjust replication factors and balance partitions. > > -Todd > > > On Fri, Dec 22, 2017 at 9:21 AM, Sagar <sagarmeansoc...@gmail.com> wrote: > > > Hi Todd, > > > > Thanks for the reply. Problem is I have about 160 topics(5 partitions for > > each) for which I need to increase the replication factors for. So, I > would > > have to find the current leader for each of the partitions and hand code > > the json which would become tedious. > > > > The partition leader info is stored in zookeeper? If yes, then can I > > probably use that get the current json and then build the json which can > be > > fed to the tool. I tried to search using the zk shell but couldn't > find... > > > > Sagar. > > > > On Fri, Dec 22, 2017 at 7:45 PM, Todd Palino <tpal...@gmail.com> wrote: > > > > > Preferred replica election is naive. It will always follow the order of > > the > > > replicas as they are set. So if you want to set the default leader, > just > > > make it the first replica in the list for the partition. We build the > > JASON > > > this way all the time. > > > > > > -Todd > > > > > > > > > On Dec 22, 2017 6:46 AM, "Sagar" <sagarmeansoc...@gmail.com> wrote: > > > > > > Hi, > > > > > > Had a question on Kafka reassign partitions tool. > > > > > > We have a 3 node cluster but our replication factor is set to 1. So we > > have > > > been looking to increase it to 3 for HA. > > > > > > I tried the tool on a couple of topics and it increases the replication > > > factor alright. Also it doesn't change the leader as the leader still > is > > in > > > the RAR. > > > > > > This is how I run it: > > > > > > Json which is used: > > > > > > {"version":1,"partitions":[ > > > > > > {"topic":"cric-engine.engine.eng_fow","partition":3," > > > replicas":[92,51,101]}]} > > > > > > Earlier config for the topic > > > > > > kafka-topics --describe --topic cric-engine.engine.eng_fow --zookeeper > > > 10.0.4.165:2181,10.0.5.139:2181,10.0.6.106:2181 > > > Topic:cric-engine.engine.eng_fow PartitionCount:5 ReplicationFactor:3 > > > Configs: > > > Topic: cric-engine.engine.eng_fow Partition: 0 Leader: 101 Replicas: > > > 92,51,101 Isr: 101,51,92 > > > Topic: cric-engine.engine.eng_fow Partition: 1 Leader: 51 Replicas: > > > 92,51,101 Isr: 51,101,92 > > > Topic: cric-engine.engine.eng_fow Partition: 2 Leader: 92 Replicas: 92 > > Isr: > > > 92 > > > Topic: cric-engine.engine.eng_fow Partition: 3 Leader: 101 Replicas: > 101 > > > Isr: > > > 101 > > > Topic: cric-engine.engine.eng_fow Partition: 4 Leader: 51 Replicas: 51 > > Isr: > > > 51 > > > > > > After running: > > > > > > kafka-reassign-partitions --reassignment-json-file > > > increase-replication-factor.json --execute --zookeeper 10.0.4.165:2181 > , > > > 10.0.5.139:2181,10.0.6.106:2181 > > > > > > partitions 3 Replicas increase: > > > > > > kafka-topics --describe --topic cric-engine.engine.eng_fow --zookeeper > > > 10.0.4.165:2181,10.0.5.139:2181,10.0.6.106:2181 > > > Topic:cric-engine.engine.eng_fow PartitionCount:5 ReplicationFactor:3 > > > Configs: > > > Topic: cric-engine.engine.eng_fow Partition: 0 Leader: 101 Replicas: > > > 92,51,101 Isr: 101,51,92 > > > Topic: cric-engine.engine.eng_fow Partition: 1 Leader: 51 Replicas: > > > 92,51,101 Isr: 51,101,92 > > > Topic: cric-engine.engine.eng_fow Partition: 2 Leader: 92 Replicas: 92 > > Isr: > > > 92 > > > Topic: cric-engine.engine.eng_fow Partition: 3 Leader: 101 Replicas: > > > 92,51,101 Isr: 101,51,92 > > > Topic: cric-engine.engine.eng_fow Partition: 4 Leader: 51 Replicas: 51 > > Isr: > > > 51 > > > > > > What I wanted to know is that does it affect the preferred replica? If > > you > > > see the Replicas, all of them are now 92,51,101 even though the leader > > has > > > remained the same from before. So, if any of the broker goes down or we > > > run kafka-preferred-replica-election.sh, wouldn't it move all the > > leaders > > > to broker 92? Is my assesment correct? > > > > > > If yes, then is there a way I can still do this operation by getting > > leader > > > for a partition first, then adding it to the replica list and then > > building > > > the json dynamically? > > > > > > Thanks! > > > Sagar. > > > > > > > > > -- > *Todd Palino* > Senior Staff Engineer, Site Reliability > Data Infrastructure Streaming > > > > linkedin.com/in/toddpalino >
