Hi,

I have an even number of kafka brokers running on AWS, across two different
AZs. What I am observing is that for all topics, sometimes both the leader
and replica of a partition are present in the same AZ. I am looking to
spread that across two different AZs. The ReplicationFactor for all topics
is set to 2.

For example,

# ./kafka-topics.sh --bootstrap-server localhost:9092 --describe --topic
my-topic
Topic: my-topic TopicId: abcabcabcabc PartitionCount: 12 ReplicationFactor:
2 Configs: segment.bytes=1073741824,max.message.bytes=10485760
Topic: my-topic Partition: 0 Leader: 31 Replicas: 31,32 Isr: 31,32
Topic: my-topic Partition: 1 Leader: 32 Replicas: 32,11 Isr: 11,32
Topic: my-topic Partition: 2 Leader: 11 Replicas: 11,12 Isr: 11,12
Topic: my-topic Partition: 3 Leader: 12 Replicas: 12,31 Isr: 12,31
Topic: my-topic Partition: 4 Leader: 31 Replicas: 31,11 Isr: 11,31
Topic: my-topic Partition: 5 Leader: 32 Replicas: 32,12 Isr: 12,32
Topic: my-topic Partition: 6 Leader: 11 Replicas: 11,31 Isr: 11,31
Topic: my-topic Partition: 7 Leader: 12 Replicas: 12,32 Isr: 12,32
Topic: my-topic Partition: 8 Leader: 31 Replicas: 31,12 Isr: 12,31
Topic: my-topic Partition: 9 Leader: 32 Replicas: 32,31 Isr: 31,32
Topic: my-topic Partition: 10 Leader: 11 Replicas: 11,32 Isr: 11,32
Topic: my-topic Partition: 11 Leader: 12 Replicas: 12,11 Isr: 11,12

Here partitions 0, 2, 9, 11 are not spread across two AZs.

 So my goal is to know whether there is any knob by which I can force the
leader and replica to go to different AZs and is there any automated
way/tool to handle this for existing partitions. In other words, I want the
leader and replica to be distributed across different AZs now and in future.

The end goal is to use "replica.selector.class =
org.apache.kafka.common.replica.RackAwareReplicaSelector" and then KIP 392
to read from the closest replica when the consumers are in kubernetes pods
across different AZs. And this will in turn reduce cross AZ traffic and
subsequently AWS bill.

Thanks,

Reply via email to