Hello, We're currently testing Mirrormaker 2.0 functionality for replication between clusters. I have successfully run the Mirrormaker 2.0 script (connect-mirror-maker.sh) using this config, replicating between two Kubernetes Kafka broker instances: Clusters = MC,DC MC.bootstrap.servers=172.17.5.249:32010 DC.bootstrap.servers=172.17.5.254:32000 topics=test-topic MC->DC.topics=test-topic MC->DC.enabled=true replication.factor=1 checkpoints.topic.replication.factor=1 heartbeats.topic.replication.factor=1 offset-syncs.topic.replication.factor=1 offset.storage.replication.factor=1 status.storage.replication.factor=1 config.storage.replication.factor=1
So the next step is to roll MirrorMaker 2.0 into our Java service architecture. To do this I follow the test examples (by Ryanne Dolan) and create an instance of the MirrorMakerConfig class as a container for the config and then instantiate a MirrorMaker class itself: //create the map for the mirrormaker 2.0 properties HashMap<String,String> mm2Props = new HashMap<>(); //TODO: remove this manual props mm2Props.put("clusters","MC,DC"); mm2Props.put("MC.bootstrap.servers","172.17.5.249:32010"); mm2Props.put("DC.bootstrap.servers","172.17.5.254:32000"); mm2Props.put("MC->DC.topics","test-topic"); mm2Props.put("MC->DC.enabled","true"); mm2Props.put("sync.topic.acls.enabled", "false"); mm2Props.put("emit.checkpoints.interval.seconds", "1"); mm2Props.put("emit.heartbeats.interval.seconds", "1"); mm2Props.put("refresh.topics.interval.seconds", "1"); mm2Props.put("refresh.groups.interval.seconds", "1"); mm2Props.put("checkpoints.topic.replication.factor", "1"); mm2Props.put("heartbeats.topic.replication.factor", "1"); mm2Props.put("offset-syncs.topic.replication.factor", "1"); mm2Props.put("config.storage.topic.replication.factor", "1"); mm2Props.put("offset.storage.topic.replication.factor", "1"); mm2Props.put("status.storage.topic.replication.factor", "1"); mm2Props.put("replication.factor", "1"); //instantiate our new Mirrormaker 2.0 config object MirrorMakerConfig mm2Config = new MirrorMakerConfig(mm2Props); //instantiate Mirrormaker 2.0 and run this.mm2 = new MirrorMaker(mm2Config,null,Time.SYSTEM); this.mm2.start(); Looking at the logs, this works up to the point where the internal MM2 topics are to be created (they demand replication factor of 3 even though the config specifies 1): 2020-02-20 13:40:42.518 INFO 7 --- [erder-connect-2] o.a.k.clients.admin.AdminClientConfig : AdminClientConfig values: bootstrap.servers = [172.17.5.254:32000] client.dns.lookup = default client.id = connections.max.idle.ms = 300000 metadata.max.age.ms = 300000 metric.reporters = [] metrics.num.samples = 2 metrics.recording.level = INFO metrics.sample.window.ms = 30000 receive.buffer.bytes = 65536 reconnect.backoff.max.ms = 1000 reconnect.backoff.ms = 50 request.timeout.ms = 120000 retries = 5 retry.backoff.ms = 100 sasl.client.callback.handler.class = null sasl.jaas.config = null sasl.kerberos.kinit.cmd = /usr/bin/kinit sasl.kerberos.min.time.before.relogin = 60000 sasl.kerberos.service.name = null sasl.kerberos.ticket.renew.jitter = 0.05 sasl.kerberos.ticket.renew.window.factor = 0.8 sasl.login.callback.handler.class = null sasl.login.class = null sasl.login.refresh.buffer.seconds = 300 sasl.login.refresh.min.period.seconds = 60 sasl.login.refresh.window.factor = 0.8 sasl.login.refresh.window.jitter = 0.05 sasl.mechanism = GSSAPI security.protocol = PLAINTEXT send.buffer.bytes = 131072 ssl.cipher.suites = null ssl.enabled.protocols = [TLSv1.2, TLSv1.1, TLSv1] ssl.endpoint.identification.algorithm = https ssl.key.password = null ssl.keymanager.algorithm = SunX509 ssl.keystore.location = null ssl.keystore.password = null ssl.keystore.type = JKS ssl.protocol = TLS ssl.provider = null ssl.secure.random.implementation = null ssl.trustmanager.algorithm = PKIX ssl.truststore.location = null ssl.truststore.password = null ssl.truststore.type = JKS 2020-02-20 13:40:42.517 WARN 7 --- [erder-connect-1] o.a.k.clients.admin.AdminClientConfig : The configuration 'producer.bootstrap.servers' was supplied but isn't a known config. 2020-02-20 13:40:42.521 WARN 7 --- [erder-connect-1] o.a.k.clients.admin.AdminClientConfig : The configuration 'group.id' was supplied but isn't a known config. 2020-02-20 13:40:42.521 WARN 7 --- [erder-connect-1] o.a.k.clients.admin.AdminClientConfig : The configuration 'status.storage.topic' was supplied but isn't a known config. 2020-02-20 13:40:42.521 WARN 7 --- [erder-connect-1] o.a.k.clients.admin.AdminClientConfig : The configuration 'offset.storage.topic.replication.factor' was supplied but isn't a known config. 2020-02-20 13:40:42.521 WARN 7 --- [erder-connect-1] o.a.k.clients.admin.AdminClientConfig : The configuration 'header.converter' was supplied but isn't a known config. 2020-02-20 13:40:42.521 WARN 7 --- [erder-connect-1] o.a.k.clients.admin.AdminClientConfig : The configuration 'consumer.bootstrap.servers' was supplied but isn't a known config. 2020-02-20 13:40:42.521 WARN 7 --- [erder-connect-1] o.a.k.clients.admin.AdminClientConfig : The configuration 'config.storage.topic.replication.factor' was supplied but isn't a known config. 2020-02-20 13:40:42.521 WARN 7 --- [erder-connect-1] o.a.k.clients.admin.AdminClientConfig : The configuration 'offset.storage.topic' was supplied but isn't a known config. 2020-02-20 13:40:42.521 WARN 7 --- [erder-connect-1] o.a.k.clients.admin.AdminClientConfig : The configuration 'value.converter' was supplied but isn't a known config. 2020-02-20 13:40:42.522 WARN 7 --- [erder-connect-1] o.a.k.clients.admin.AdminClientConfig : The configuration 'status.storage.topic.replication.factor' was supplied but isn't a known config. 2020-02-20 13:40:42.522 WARN 7 --- [erder-connect-1] o.a.k.clients.admin.AdminClientConfig : The configuration 'key.converter' was supplied but isn't a known config. 2020-02-20 13:40:42.522 WARN 7 --- [erder-connect-1] o.a.k.clients.admin.AdminClientConfig : The configuration 'admin.bootstrap.servers' was supplied but isn't a known config. 2020-02-20 13:40:42.522 INFO 7 --- [erder-connect-1] o.a.kafka.common.utils.AppInfoParser : Kafka version: 2.3.0 2020-02-20 13:40:42.522 INFO 7 --- [erder-connect-1] o.a.kafka.common.utils.AppInfoParser : Kafka commitId: fc1aaa116b661c8a 2020-02-20 13:40:42.522 INFO 7 --- [erder-connect-1] o.a.kafka.common.utils.AppInfoParser : Kafka startTimeMs: 1582206042522 2020-02-20 13:40:42.534 WARN 7 --- [erder-connect-2] o.a.k.clients.admin.AdminClientConfig : The configuration 'config.storage.topic' was supplied but isn't a known config. 2020-02-20 13:40:42.539 WARN 7 --- [erder-connect-2] o.a.k.clients.admin.AdminClientConfig : The configuration 'producer.bootstrap.servers' was supplied but isn't a known config. 2020-02-20 13:40:42.539 WARN 7 --- [erder-connect-2] o.a.k.clients.admin.AdminClientConfig : The configuration 'group.id' was supplied but isn't a known config. 2020-02-20 13:40:42.539 WARN 7 --- [erder-connect-2] o.a.k.clients.admin.AdminClientConfig : The configuration 'status.storage.topic' was supplied but isn't a known config. 2020-02-20 13:40:42.539 WARN 7 --- [erder-connect-2] o.a.k.clients.admin.AdminClientConfig : The configuration 'offset.storage.topic.replication.factor' was supplied but isn't a known config. 2020-02-20 13:40:42.539 WARN 7 --- [erder-connect-2] o.a.k.clients.admin.AdminClientConfig : The configuration 'header.converter' was supplied but isn't a known config. 2020-02-20 13:40:42.540 WARN 7 --- [erder-connect-2] o.a.k.clients.admin.AdminClientConfig : The configuration 'consumer.bootstrap.servers' was supplied but isn't a known config. 2020-02-20 13:40:42.540 WARN 7 --- [erder-connect-2] o.a.k.clients.admin.AdminClientConfig : The configuration 'config.storage.topic.replication.factor' was supplied but isn't a known config. 2020-02-20 13:40:42.540 WARN 7 --- [erder-connect-2] o.a.k.clients.admin.AdminClientConfig : The configuration 'offset.storage.topic' was supplied but isn't a known config. 2020-02-20 13:40:42.540 WARN 7 --- [erder-connect-2] o.a.k.clients.admin.AdminClientConfig : The configuration 'value.converter' was supplied but isn't a known config. 2020-02-20 13:40:42.541 WARN 7 --- [erder-connect-2] o.a.k.clients.admin.AdminClientConfig : The configuration 'status.storage.topic.replication.factor' was supplied but isn't a known config. 2020-02-20 13:40:42.541 WARN 7 --- [erder-connect-2] o.a.k.clients.admin.AdminClientConfig : The configuration 'key.converter' was supplied but isn't a known config. 2020-02-20 13:40:42.541 WARN 7 --- [erder-connect-2] o.a.k.clients.admin.AdminClientConfig : The configuration 'admin.bootstrap.servers' was supplied but isn't a known config. 2020-02-20 13:40:42.541 INFO 7 --- [erder-connect-2] o.a.kafka.common.utils.AppInfoParser : Kafka version: 2.3.0 2020-02-20 13:40:42.543 INFO 7 --- [erder-connect-2] o.a.kafka.common.utils.AppInfoParser : Kafka commitId: fc1aaa116b661c8a 2020-02-20 13:40:42.543 INFO 7 --- [ scheduling-1] o.a.kafka.connect.mirror.MirrorMaker : Kafka MirrorMaker started 2020-02-20 13:40:42.543 INFO 7 --- [erder-connect-2] o.a.kafka.common.utils.AppInfoParser : Kafka startTimeMs: 1582206042541 2020-02-20 13:40:42.572 ERROR 7 --- [erder-connect-1] o.a.k.c.r.distributed.DistributedHerder : [Worker clientId=connect-1, groupId=DC-mm2] Uncaught exception in herder work thread, exiting: org.apache.kafka.connect.errors.ConnectException: Error while attempting to create/find topic(s) 'mm2-offsets.DC.internal' at org.apache.kafka.connect.util.TopicAdmin.createTopics(TopicAdmin.java:262) ~[connect-runtime-2.3.0.jar!/:na] at org.apache.kafka.connect.storage.KafkaOffsetBackingStore$1.run(KafkaOffsetBackingStore.java:99) ~[connect-runtime-2.3.0.jar!/:na] at org.apache.kafka.connect.util.KafkaBasedLog.start(KafkaBasedLog.java:127) ~[connect-runtime-2.3.0.jar!/:na] at org.apache.kafka.connect.storage.KafkaOffsetBackingStore.start(KafkaOffsetBackingStore.java:109) ~[connect-runtime-2.3.0.jar!/:na] at org.apache.kafka.connect.runtime.Worker.start(Worker.java:184) ~[connect-runtime-2.3.0.jar!/:na] at org.apache.kafka.connect.runtime.AbstractHerder.startServices(AbstractHerder.java:121) ~[connect-runtime-2.3.0.jar!/:na] at org.apache.kafka.connect.runtime.distributed.DistributedHerder.run(DistributedHerder.java:240) ~[connect-runtime-2.3.0.jar!/:na] at java.base/java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:515) ~[na:na] at java.base/java.util.concurrent.FutureTask.run(FutureTask.java:264) ~[na:na] at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128) ~[na:na] at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628) ~[na:na] at java.base/java.lang.Thread.run(Thread.java:834) ~[na:na] Caused by: java.util.concurrent.ExecutionException: org.apache.kafka.common.errors.InvalidReplicationFactorException: Replication factor: 3 larger than available brokers: 1. at org.apache.kafka.common.internals.KafkaFutureImpl.wrapAndThrow(KafkaFutureImpl.java:45) ~[kafka-clients-2.3.0.jar!/:na] at org.apache.kafka.common.internals.KafkaFutureImpl.access$000(KafkaFutureImpl.java:32) ~[kafka-clients-2.3.0.jar!/:na] at org.apache.kafka.common.internals.KafkaFutureImpl$SingleWaiter.await(KafkaFutureImpl.java:89) ~[kafka-clients-2.3.0.jar!/:na] at org.apache.kafka.common.internals.KafkaFutureImpl.get(KafkaFutureImpl.java:260) ~[kafka-clients-2.3.0.jar!/:na] at org.apache.kafka.connect.util.TopicAdmin.createTopics(TopicAdmin.java:229) ~[connect-runtime-2.3.0.jar!/:na] ... 11 common frames omitted Caused by: org.apache.kafka.common.errors.InvalidReplicationFactorException: Replication factor: 3 larger than available brokers: 1. Any help would be appreciated. Thanks Pirow Engelbrecht | Senior Design Engineer Tel +27 12 678 9740 (ext. 9879) | Cell +27 63 148 3376 76 Regency Drive | Irene | Centurion | 0157<https://goo.gl/maps/v9ZbwjqpPyL2> [cid:image001.jpg@01D5E804.4664EBF0]<https://etion.co.za> Facebook<https://www.facebook.com/Etion-Limited-2194612947433812?_rdc=1&_rdr> | YouTube<https://www.youtube.com/channel/UCUY-5oeACtLk2uTsEjZCU6A> | LinkedIn<https://www.linkedin.com/company/etionltd> | Twitter<https://twitter.com/Etionlimited> | Instagram<https://www.instagram.com/Etionlimited/>