Dear Kafka Users, I would like to use Kafka 0.8.x in a multi-cluster environment so that when my primary cluster fails, producers and consumers could switch to the secondary cluster. Clusters would be hosted in different data centers.
A possibility would be mirroring topics (similar to Kafka 0.7.x mirror maker). The issue with this is consumer offset management, since a mirrored message will probably have different offset than the source message. Running a single Kafka cluster with nodes in both data centers raises the question of how to ensure a message was persisted by at least one broker in each data center. Even with all in sync replicas ACK requested, the producer can't be sure what brokers persisted a message because in sync replicas might change dynamically. Could you please share your experience about running Kafka 0.8.x cluster(s) on multiple data centers? Thank you very much. Best regards, Janos
