And can you share the patch... On Sun, Dec 22, 2019 at 10:34 PM Vishal Santoshi <vishal.santo...@gmail.com> wrote:
> We also have a large number of topics 1500 plus and in a cross DC > replication. How do we increase the default timeouts ? > > > On Wed, Dec 11, 2019 at 2:26 PM Ryanne Dolan <ryannedo...@gmail.com> > wrote: > >> Hey Peter. Do you see any timeouts in the logs? The internal scheduler >> will >> timeout each task after 60 seconds by default, which might not be long >> enough to finish some of the bootstrap tasks in your case. My team has >> observed that behavior in super-flaky environments, e.g. when connectivity >> drops during bootstrapping, in which case MirrorSourceConnector can get >> into a funky state. This resolves when it refreshes its state after a >> while. The default refresh interval of 10 minutes seems to jibe with your >> observations. >> >> My team patched our internal MM2 build months ago to force bootstrapping >> to >> complete correctly. I can share the patch, and if it helps we can raise a >> PR. >> >> Ryanne >> >> On Mon, Dec 9, 2019 at 5:28 AM Péter Sinóros-Szabó >> <peter.sinoros-sz...@transferwise.com.invalid> wrote: >> >> > Hi, >> > >> > I am experimenting with Mirror Make 2 in 2.4.0-rc3. It seems to start up >> > fine, connects to both source and destination, creates new topics... >> > But it does not start to actually mirror the messages until about 12 >> > minutes after MM2 was started. I would expect it to start mirroring in >> some >> > seconds after startup. >> > >> > Source cluster has about 2800 partitions, destination cluster is empty. >> > Both clusters are in AWS but in different regions. >> > >> > What may cause the 12 minutes delay? >> > >> > Config is: >> > --- >> > clusters = eucmain, euwbackup >> > eucmain.bootstrap.servers = >> > test-kafka-main-fra01.xx:9092,test-kafka-main-fra02.xx:9092 >> > euwbackup.bootstrap.servers = 172.30.197.203:9092,172.30.213.104:9092 >> > eucmain->euwbackup.enabled = true >> > eucmain->euwbackup.topics = .* >> > eucmain->euwbackup.topics.blacklist = ^(kafka|kmf|__).* >> > eucmain->euwbackup.rename.topics = false >> > replication.policy.separator = __ >> > eucmain.client.id = mm2 >> > >> > I do not see any serious errors in the logs that I would think of a >> cause >> > of this. >> > >> > Thanks, >> > Peter >> > >> >
