Hi, I have a similar issue. I changed the source cluster bootstrap address and MM2 picked it up only partially. Some parts of it still use the old address, some the new. The old and the new address list is routed to the same cluster, same brokers, just on a different network path.
So is there any way to force the configuration update? Cheers, Peter On Wed, 4 Nov 2020 at 18:39, Ning Zhang <ning2008w...@gmail.com> wrote: > if your new topics are not named "topic1" or "topic2", maybe you want to > use regex * to allow more topics to be considered by Mm2 > > # regex which defines which topics gets replicated. For eg "foo-.*" > src-cluster->dst-cluster.topics = topic1,topic2 > > On 2020/10/30 01:48:00, "Devaki, Srinivas" <me@eightnoteight.space> > wrote: > > Hi Folks, > > > > I'm running mirror maker as a dedicated cluster as given in the > > mirrormaker 2 doc. but for some reason when I add new topics and > > deploy the mirror maker it's not detecting the new topics at all, even > > the config dumps in the mirror maker startup logs don't show the newly > > added topics. > > > > I've attached the config that I'm using, initially I assumed that > > there might be some refresh configuration option either in connect or > > mirror maker, but the connect rest api doesn't seem to be working in > > this mode and also couldn't find any refresh configuration option. > > > > Any ideas on this? Thank you in advance > > > > ``` > > clusters = src-cluster, dst-cluster > > > > # disable topic prefixes > > src-cluster.replication.policy.separator = > > dst-cluster.replication.policy.separator = > > replication.policy.separator = > > source.cluster.alias = > > target.cluster.alias = > > > > > > # enable idemptotence > > source.cluster.producer.enable.idempotence = true > > target.cluster.producer.enable.idempotence = true > > > > # connection information for each cluster > > # This is a comma separated host:port pairs for each cluster > > # for e.g. "A_host1:9092, A_host2:9092, A_host3:9092" > > src-cluster.bootstrap.servers = > > > sng-kfnode1.internal:9092,sng-kfnode1.internal:9092,sng-kfnode1.internal:9092 > > dst-cluster.bootstrap.servers = > > > prod-online-v2-kafka-1.internal:9092,prod-online-v2-kafka-2.internal:9092,prod-online-v2-kafka-3.internal:9092,prod-online-v2-kafka-4.internal:9092,prod-online-v2-kafka-5.internal:9092 > > > > # regex which defines which topics gets replicated. For eg "foo-.*" > > src-cluster->dst-cluster.topics = topic1,topic2 > > > > # client-id > > src-cluster.client.id = prod-mm2-onlinev1-to-onlinev2-consumer-v0 > > dst-cluster.client.id = prod-mm2-onlinev1-to-onlinev2-producer-v0 > > > > > > # group.instance.id=_mirror_make_instance_1 > > # consumer should periodically emit heartbeats > > src-cluster->dst-cluster.consumer.auto.offset.reset = earliest > > src-cluster->dst-cluster.consumer.overrides.auto.offset.reset = earliest > > > > # connector should periodically emit heartbeats > > src-cluster->dst-cluster.emit.heartbeats.enabled = true > > > > # frequency of heartbeats, default is 5 seconds > > src-cluster->dst-cluster.emit.heartbeats.interval.seconds = 10 > > > > # connector should periodically emit consumer offset information > > src-cluster->dst-cluster.emit.checkpoints.enabled = true > > > > # frequency of checkpoints, default is 5 seconds > > src-cluster->dst-cluster.emit.checkpoints.interval.seconds = 10 > > > > # whether to monitor source cluster ACLs for changes > > src-cluster->dst-cluster.sync.topic.acls.enabled = false > > > > # whether or not to monitor source cluster for configuration changes > > src-cluster->dst-cluster.sync.topic.configs.enabled = true > > # add retention.ms to the default list given in the > DefaultConfigPropertyFilter > > # > https://github.com/apache/kafka/blob/889fd31b207b86db6d059792131d14389639d9e4/connect/mirror/src/main/java/org/apache/kafka/connect/mirror/DefaultConfigPropertyFilter.java#L33-L38 > > src-cluster->dst-cluster.config.properties.blacklist = > > follower\\.replication\\.throttled\\.replicas, \ > > > > leader\\.replication\\.throttled\\.replicas, \ > > > > message\\.timestamp\\.difference\\.max\\.ms, \ > > > message\\.timestamp\\.type, \ > > > > unclean\\.leader\\.election\\.enable, \ > > > min\\.insync\\.replicas, \ > > retention\\.ms > > > > # connector should periodically check for new topics > > src-cluster->dst-cluster.refresh.topics.enabled = true > > > > # frequency to check source cluster for new topics, default is 5 seconds > > src-cluster->dst-cluster.refresh.topics.interval.seconds = 300 > > > > # enable and configure individual replication flows > > src-cluster->dst-cluster.enabled = true > > dst-cluster->src-cluster.enabled = false > > > > > > # Setting replication factor of newly created remote topics > > # replication.factor=2 > > > > ############################# Internal Topic Settings > > ############################# > > # The replication factor for mm2 internal topics "heartbeats", > > "B.checkpoints.internal" and > > # "mm2-offset-syncs.B.internal" > > # For anything other than development testing, a value greater than 1 > > is recommended to ensure availability such as 3. > > checkpoints.topic.replication.factor=3 > > # 14 days > > checkpoints.topic.retention.ms=1209600000 > > heartbeats.topic.replication.factor=3 > > offset-syncs.topic.replication.factor=3 > > > > # The replication factor for connect internal topics > > "mm2-configs.B.internal", "mm2-offsets.B.internal" and > > # "mm2-status.B.internal" > > # For anything other than development testing, a value greater than 1 > > is recommended to ensure availability such as 3. > > offset.storage.replication.factor=3 > > status.storage.replication.factor=3 > > config.storage.replication.factor=3 > > > > # customize as needed > > # replication.policy.separator = _ > > # sync.topic.acls.enabled = false > > # emit.heartbeats.interval.seconds = 5 > > ``` > > > > Thanks > > >
