Hi, I have an even number of kafka brokers running on AWS, across two different AZs. What I am observing is that for all topics, sometimes both the leader and replica of a partition are present in the same AZ. I am looking to spread that across two different AZs. The ReplicationFactor for all topics is set to 2.
For example, # ./kafka-topics.sh --bootstrap-server localhost:9092 --describe --topic my-topic Topic: my-topic TopicId: abcabcabcabc PartitionCount: 12 ReplicationFactor: 2 Configs: segment.bytes=1073741824,max.message.bytes=10485760 Topic: my-topic Partition: 0 Leader: 31 Replicas: 31,32 Isr: 31,32 Topic: my-topic Partition: 1 Leader: 32 Replicas: 32,11 Isr: 11,32 Topic: my-topic Partition: 2 Leader: 11 Replicas: 11,12 Isr: 11,12 Topic: my-topic Partition: 3 Leader: 12 Replicas: 12,31 Isr: 12,31 Topic: my-topic Partition: 4 Leader: 31 Replicas: 31,11 Isr: 11,31 Topic: my-topic Partition: 5 Leader: 32 Replicas: 32,12 Isr: 12,32 Topic: my-topic Partition: 6 Leader: 11 Replicas: 11,31 Isr: 11,31 Topic: my-topic Partition: 7 Leader: 12 Replicas: 12,32 Isr: 12,32 Topic: my-topic Partition: 8 Leader: 31 Replicas: 31,12 Isr: 12,31 Topic: my-topic Partition: 9 Leader: 32 Replicas: 32,31 Isr: 31,32 Topic: my-topic Partition: 10 Leader: 11 Replicas: 11,32 Isr: 11,32 Topic: my-topic Partition: 11 Leader: 12 Replicas: 12,11 Isr: 11,12 Here partitions 0, 2, 9, 11 are not spread across two AZs. So my goal is to know whether there is any knob by which I can force the leader and replica to go to different AZs and is there any automated way/tool to handle this for existing partitions. In other words, I want the leader and replica to be distributed across different AZs now and in future. The end goal is to use "replica.selector.class = org.apache.kafka.common.replica.RackAwareReplicaSelector" and then KIP 392 to read from the closest replica when the consumers are in kubernetes pods across different AZs. And this will in turn reduce cross AZ traffic and subsequently AWS bill. Thanks,
