Dan created KAFKA-5115:
--------------------------
Summary: Use bootstrap.servers to refresh metadata
Key: KAFKA-5115
URL: https://issues.apache.org/jira/browse/KAFKA-5115
Project: Kafka
Issue Type: Improvement
Affects Versions: 0.10.2.0
Reporter: Dan
Currently, it seems that bootstrap.servers list is used only when the producer
starts, to discover the cluster, and subsequent metadata refreshes go to the
discovered brokers directly.
We would like to use the bootstrap.servers list for metadata refresh to support
a failover mechanism by providing a VIP which can dynamically redirect requests
to a secondary Kafka cluster if the primary is down.
Consider the following use case, where "kafka-cluster.local" is a VIP on a load
balancer with priority server pools that point to two different Kafka clusters
(so when all servers of cluster #1 are down, it automatically redirects to
servers from cluster #2).
bootstrap.servers: kafka-cluster.local:9092
1) Producer starts, connects to kafka-cluster.local and discovers all servers
from cluster #1
2) Producer starts producing to cluster #1
3) cluster #1 goes down
4) Producer detects the failure, refreshes metadata from kafka-cluster.local
(which now returns nodes from cluster #2)
5) Producer starts producing to cluster #2
6) cluster #1 is brought back online, and kafka-cluster.local now points to it
again
In the current state, it seems that the producer will never revert to cluster
#1 because it continues to refresh its metadata from the brokers of cluster #2,
even though kafka-cluster.local no longer points to that cluster.
If we could force the metadata refresh to happen against "kafka-cluster.local",
it would enable automatic failover and failback between the clusters.
--
This message was sent by Atlassian JIRA
(v6.3.15#6346)
