Hi Folks, I've also ran a console-consumer on the `mm2-configs` kafka topic created by the mirror maker and found that even after the restart of the mirror maker 2 with new config, the config registered in the mm2-configs kafka topic is still pointing to a legacy mirror maker configuration.
Thanks On Fri, Oct 30, 2020 at 7:18 AM Devaki, Srinivas <me@eightnoteight.space> wrote: > > Hi Folks, > > I'm running mirror maker as a dedicated cluster as given in the > mirrormaker 2 doc. but for some reason when I add new topics and > deploy the mirror maker it's not detecting the new topics at all, even > the config dumps in the mirror maker startup logs don't show the newly > added topics. > > I've attached the config that I'm using, initially I assumed that > there might be some refresh configuration option either in connect or > mirror maker, but the connect rest api doesn't seem to be working in > this mode and also couldn't find any refresh configuration option. > > Any ideas on this? Thank you in advance > > ``` > clusters = src-cluster, dst-cluster > > # disable topic prefixes > src-cluster.replication.policy.separator = > dst-cluster.replication.policy.separator = > replication.policy.separator = > source.cluster.alias = > target.cluster.alias = > > > # enable idemptotence > source.cluster.producer.enable.idempotence = true > target.cluster.producer.enable.idempotence = true > > # connection information for each cluster > # This is a comma separated host:port pairs for each cluster > # for e.g. "A_host1:9092, A_host2:9092, A_host3:9092" > src-cluster.bootstrap.servers = > sng-kfnode1.internal:9092,sng-kfnode1.internal:9092,sng-kfnode1.internal:9092 > dst-cluster.bootstrap.servers = > prod-online-v2-kafka-1.internal:9092,prod-online-v2-kafka-2.internal:9092,prod-online-v2-kafka-3.internal:9092,prod-online-v2-kafka-4.internal:9092,prod-online-v2-kafka-5.internal:9092 > > # regex which defines which topics gets replicated. For eg "foo-.*" > src-cluster->dst-cluster.topics = topic1,topic2 > > # client-id > src-cluster.client.id = prod-mm2-onlinev1-to-onlinev2-consumer-v0 > dst-cluster.client.id = prod-mm2-onlinev1-to-onlinev2-producer-v0 > > > # group.instance.id=_mirror_make_instance_1 > # consumer should periodically emit heartbeats > src-cluster->dst-cluster.consumer.auto.offset.reset = earliest > src-cluster->dst-cluster.consumer.overrides.auto.offset.reset = earliest > > # connector should periodically emit heartbeats > src-cluster->dst-cluster.emit.heartbeats.enabled = true > > # frequency of heartbeats, default is 5 seconds > src-cluster->dst-cluster.emit.heartbeats.interval.seconds = 10 > > # connector should periodically emit consumer offset information > src-cluster->dst-cluster.emit.checkpoints.enabled = true > > # frequency of checkpoints, default is 5 seconds > src-cluster->dst-cluster.emit.checkpoints.interval.seconds = 10 > > # whether to monitor source cluster ACLs for changes > src-cluster->dst-cluster.sync.topic.acls.enabled = false > > # whether or not to monitor source cluster for configuration changes > src-cluster->dst-cluster.sync.topic.configs.enabled = true > # add retention.ms to the default list given in the > DefaultConfigPropertyFilter > # > https://github.com/apache/kafka/blob/889fd31b207b86db6d059792131d14389639d9e4/connect/mirror/src/main/java/org/apache/kafka/connect/mirror/DefaultConfigPropertyFilter.java#L33-L38 > src-cluster->dst-cluster.config.properties.blacklist = > follower\\.replication\\.throttled\\.replicas, \ > > leader\\.replication\\.throttled\\.replicas, \ > > message\\.timestamp\\.difference\\.max\\.ms, \ > > message\\.timestamp\\.type, \ > > unclean\\.leader\\.election\\.enable, \ > min\\.insync\\.replicas, \ > retention\\.ms > > # connector should periodically check for new topics > src-cluster->dst-cluster.refresh.topics.enabled = true > > # frequency to check source cluster for new topics, default is 5 seconds > src-cluster->dst-cluster.refresh.topics.interval.seconds = 300 > > # enable and configure individual replication flows > src-cluster->dst-cluster.enabled = true > dst-cluster->src-cluster.enabled = false > > > # Setting replication factor of newly created remote topics > # replication.factor=2 > > ############################# Internal Topic Settings > ############################# > # The replication factor for mm2 internal topics "heartbeats", > "B.checkpoints.internal" and > # "mm2-offset-syncs.B.internal" > # For anything other than development testing, a value greater than 1 > is recommended to ensure availability such as 3. > checkpoints.topic.replication.factor=3 > # 14 days > checkpoints.topic.retention.ms=1209600000 > heartbeats.topic.replication.factor=3 > offset-syncs.topic.replication.factor=3 > > # The replication factor for connect internal topics > "mm2-configs.B.internal", "mm2-offsets.B.internal" and > # "mm2-status.B.internal" > # For anything other than development testing, a value greater than 1 > is recommended to ensure availability such as 3. > offset.storage.replication.factor=3 > status.storage.replication.factor=3 > config.storage.replication.factor=3 > > # customize as needed > # replication.policy.separator = _ > # sync.topic.acls.enabled = false > # emit.heartbeats.interval.seconds = 5 > ``` > > Thanks
