Hi,
For higher availability and more flexible operations (upgrading, rolling ops, etc.), we are considering to federate two or more kafka clusters, each with a separated zk cluster. For example, if "topic-T" has 6 total partitions, 3 on cluster A, and another 3 on cluster B, messages will be partitioned (by key or random) into all 6 partitions ([A,0-2], [B,0-2]). On the other side, consumers have to accept two metadata broker lists for both clusters, process messages and store offsets separately. Pros: - more HA - fewer loss for single broker machine down - fewer loss when a network switch has problems or a rack is power off - do green/blue compatible upgrading, and afford crashes of one entire cluster (maybe human mistakes, maybe bugs) - with a special library of producers and consumers, we can do incompatible upgrading such as 0.7 -> 0.8 Cons: - complex operations - need wrap a special version of producers and consumers APIs for multiple clusters Do you think this is a proper solution? Does Kafka have some similar features? Thanks.
