Hi All, Has anyone been able to override the producer and consumer values of the MM2 MirrorSourceConnector as described below? Many Thanks, Jamie
-----Original Message----- From: Jamie <jamied...@aol.co.uk> To: users@kafka.apache.org <users@kafka.apache.org> Sent: Thu, 16 Sep 2021 15:52 Subject: MM2 - Overriding MirrorSourceConnector Consumer & Producer values Hi All, I've trying to override the properties of the consumer and producer in MM2 to tune them for high throughput. For example, I'm trying to set the consumers fetch.min.bytes to 100000. I'm running MM2 in a dedicated mirror maker cluster (https://cwiki.apache.org/confluence/display/KAFKA/KIP-382%3A+MirrorMaker+2.0#KIP382:MirrorMaker2.0-RunningadedicatedMirrorMakercluster) and using version 2.7.1 of Kafka. I have the equivalent of the following in my mirror maker properties file: clusters = A, B A.bootstrap.servers = A_host1:9092, A_host2:9092, A_host3:9092 B.bootstrap.servers = B_host1:9092, B_host2:9092, B_host3:9092 # enable and configure individual replication flows A->B.enabled = true # regex which defines which topics gets replicated. For eg "foo-.*" A->B.topics = .test-topic I'm trying to override the properties of the consumer which fetches records from cluster "A" and the producer that sends records to cluster "B". I've tried the following in the config file: A.consumer.fetch.min.bytes = 100000 A->B.consumer.fetch.min.bytes = 100000 A.fetch.min.bytes = 100000 B.consumer.fetch.min.bytes = 100000 B.fetch.min.bytes = 100000 None of which seem to work, when I start MM2 and go into the logs and look at the value using the the MirrorSourceConnector tasks consumer and producer config I still see the default value for fetch.min.bytes (1) being used. Am I trying to override the values of the consumer incorrectly or do I need to set these in a different place? Many Thanks, Jamie
