Hello, I have 50 on-prem field devices that each function as a single-broker kafka cluster. I am replicating to a multi-broker "master" cluster in a datacenter. I've been doing this by running the original mirrormaker in the datacenter. (Because of concerns about unauthorized data access, we cannot run mirrormaker on the field devices). This has a lot of overhead because I am running an instance of mirrormaker for every remote node. I have been testing mirrormaker2 to reduce this overhead because it supports several clusters in the configuration. However, its seems like when any of the clusters are unreachable none of the replication works. Is there a work around for this or a better way to replicate topics from several remote clusters?
Here’s an example of my mm2 properties: bootstrap.server=192.0.2.1:9092 replication.factor=2 checkpoints.topic.replication.factor=1 heartbeats.topic.replication.factor=1 offset-syncs.topic.replication.factor=1 offset.storage.replication.factor=1 status.storage.replication.factor=1 config.storage.replication.factor=1 dc-cluster.bootstrap.servers = 192.0.2.1:9092 remote-cluster1.bootstrap.servers = 192.0.2.2:9092 remote-cluster2.bootstrap.servers = 192.0.2.3:9092 clusters=dc-cluster,remote-cluster1,remote-cluster2 remote-cluster1->dc-cluster.enabled = true remote-cluster2->dc-cluster.enabled = true Thanks, Taylor Mahany
