Hello Kafka users, This is claude response for the problem mentioned. can some expert comment on this and suggest better approach?
*Problem*: I have 30 kafka brokers in my prod clusters. I want to group brokers based on the producer country. For example singapore will connext on 9092 port, India will connect on 9094 port and so on. However i have to add both port listener on all brokers. Hence I cant achive any benefit from DR point of view of migration point of view. Exmaple I want to trigger DR only for singapore and switch it off. I cant simply bring down 9092. I have to change the config and restart all brokers. What is the solution of this problem? *Response from Claude: * This is a classic listener isolation vs. broker grouping problem. The root cause is that Kafka’s listener model is per-broker, not per-group-of-brokers. The Real Solution: Broker Groups via Rack Awareness + Dedicated Listener Brokers Instead of adding all listeners to all brokers, you need to dedicate a subset of brokers per producer country/region. Architecture: Listener-Segregated Broker Groups 30 Brokers split into logical groups: Group A (Brokers 1–10) → Singapore producers → LISTENER: SG_LISTENER:9092 Group B (Brokers 11–20) → India producers → LISTENER: IN_LISTENER:9094 Group C (Brokers 21–30) → Other/Internal → LISTENER: INT_LISTENER:9096 Each broker group only has its own listener, not all listeners. Kafka Config per Group Group A (Singapore brokers): listeners=SG_LISTENER://0.0.0.0:9092 advertised.listeners=SG_LISTENER://broker-sg-01.internal:9092 listener.security.protocol.map=SG_LISTENER:SASL_SSL broker.rack=sg-az1 Group B (India brokers): listeners=IN_LISTENER://0.0.0.0:9094 advertised.listeners=IN_LISTENER://broker-in-01.internal:9094 listener.security.protocol.map=IN_LISTENER:SASL_SSL broker.rack=in-az1 DR Trigger Mechanism Now when you need to trigger DR for Singapore: |Action |What you do | |-------------------------|--------------------------------------------------------------------| |DR cutover |Update DNS / load balancer to stop routing SG producers to port 9092| |Broker isolation |Shut down brokers 1–10 only, others unaffected | |No rolling restart needed|India brokers (11–20) never had 9092 — nothing to change | |Re-onboard after DR |Bring Group A back, DNS flips back | The Missing Piece: Partition Assignment via Rack Awareness Use broker.rack + replica.selector.class so Singapore topic partitions are leader-preferenced on Group A brokers: # On all brokers replica.selector.class=org.apache.kafka.common.replica.RackAwareReplicaSelector When creating topics, set --replica-assignment explicitly or use a custom AdminClient script to pin partition leaders to the correct rack group. What About the Bootstrap Problem? Producers need to know which brokers to bootstrap from. Use separate bootstrap endpoints per region: sg-kafka.internal:9092 → resolves to brokers 1–10 only (via DNS/LB) in-kafka.internal:9094 → resolves to brokers 11–20 only This way your bootstrap.servers in producer config is region-aware, and the listener port difference reinforces the segregation. Summary of What Changes vs Your Current Design |Problem you had |Solution | |---------------------------------|--------------------------------------------------------------| |All listeners on all brokers |Assign listeners per broker group only | |DR requires restart of all 30 |DR only touches the 10 brokers in that group | |Can’t bring down one port cleanly|Bringing down Group A brokers = clean SG DR, others unaffected| |Migration risk is cluster-wide |Changes scoped to one group of 10 | The key insight is: the listener boundary should match the broker group boundary, not span the whole cluster.
