We also have a large number of topics 1500 plus and in a cross DC replication. How do we increase the default timeouts ?
On Wed, Dec 11, 2019 at 2:26 PM Ryanne Dolan <ryannedo...@gmail.com> wrote: > Hey Peter. Do you see any timeouts in the logs? The internal scheduler will > timeout each task after 60 seconds by default, which might not be long > enough to finish some of the bootstrap tasks in your case. My team has > observed that behavior in super-flaky environments, e.g. when connectivity > drops during bootstrapping, in which case MirrorSourceConnector can get > into a funky state. This resolves when it refreshes its state after a > while. The default refresh interval of 10 minutes seems to jibe with your > observations. > > My team patched our internal MM2 build months ago to force bootstrapping to > complete correctly. I can share the patch, and if it helps we can raise a > PR. > > Ryanne > > On Mon, Dec 9, 2019 at 5:28 AM Péter Sinóros-Szabó > <peter.sinoros-sz...@transferwise.com.invalid> wrote: > > > Hi, > > > > I am experimenting with Mirror Make 2 in 2.4.0-rc3. It seems to start up > > fine, connects to both source and destination, creates new topics... > > But it does not start to actually mirror the messages until about 12 > > minutes after MM2 was started. I would expect it to start mirroring in > some > > seconds after startup. > > > > Source cluster has about 2800 partitions, destination cluster is empty. > > Both clusters are in AWS but in different regions. > > > > What may cause the 12 minutes delay? > > > > Config is: > > --- > > clusters = eucmain, euwbackup > > eucmain.bootstrap.servers = > > test-kafka-main-fra01.xx:9092,test-kafka-main-fra02.xx:9092 > > euwbackup.bootstrap.servers = 172.30.197.203:9092,172.30.213.104:9092 > > eucmain->euwbackup.enabled = true > > eucmain->euwbackup.topics = .* > > eucmain->euwbackup.topics.blacklist = ^(kafka|kmf|__).* > > eucmain->euwbackup.rename.topics = false > > replication.policy.separator = __ > > eucmain.client.id = mm2 > > > > I do not see any serious errors in the logs that I would think of a cause > > of this. > > > > Thanks, > > Peter > > >
