Hey Peter. Do you see any timeouts in the logs? The internal scheduler will timeout each task after 60 seconds by default, which might not be long enough to finish some of the bootstrap tasks in your case. My team has observed that behavior in super-flaky environments, e.g. when connectivity drops during bootstrapping, in which case MirrorSourceConnector can get into a funky state. This resolves when it refreshes its state after a while. The default refresh interval of 10 minutes seems to jibe with your observations.
My team patched our internal MM2 build months ago to force bootstrapping to complete correctly. I can share the patch, and if it helps we can raise a PR. Ryanne On Mon, Dec 9, 2019 at 5:28 AM Péter Sinóros-Szabó <peter.sinoros-sz...@transferwise.com.invalid> wrote: > Hi, > > I am experimenting with Mirror Make 2 in 2.4.0-rc3. It seems to start up > fine, connects to both source and destination, creates new topics... > But it does not start to actually mirror the messages until about 12 > minutes after MM2 was started. I would expect it to start mirroring in some > seconds after startup. > > Source cluster has about 2800 partitions, destination cluster is empty. > Both clusters are in AWS but in different regions. > > What may cause the 12 minutes delay? > > Config is: > --- > clusters = eucmain, euwbackup > eucmain.bootstrap.servers = > test-kafka-main-fra01.xx:9092,test-kafka-main-fra02.xx:9092 > euwbackup.bootstrap.servers = 172.30.197.203:9092,172.30.213.104:9092 > eucmain->euwbackup.enabled = true > eucmain->euwbackup.topics = .* > eucmain->euwbackup.topics.blacklist = ^(kafka|kmf|__).* > eucmain->euwbackup.rename.topics = false > replication.policy.separator = __ > eucmain.client.id = mm2 > > I do not see any serious errors in the logs that I would think of a cause > of this. > > Thanks, > Peter >
