Hi,
You need to set refresh.topics.interval.seconds, which is 10 minutes by
default.
Ivan
On Tue, 17 Mar 2020 at 17:45, Péter Sinóros-Szabó
<peter.sinoros-sz...@transferwise.com.invalid> wrote:
> Hey,
>
> Running a MM2 cluster to mirror from A->B clusters I noticed that when I
> add a new topic to A cluster, MM2 will notice it:
> [2020-03-17 13:14:05,477] INFO Found 2719 topic-partitions on main. 1 are
> new. 0 were removed. Previously had 2718.
> (org.apache.kafka.connect.mirror.MirrorSourceConnector)
>
> That's fine. It seems that MM2 connectors just simply restart to start up
> with the new configuration.
> My problem is that it takes about 5 minutes for MM2 to start to mirror
> messages again.
>
> What I see in the logs are:
> [2020-03-17 13:14:07,107] INFO Kafka startTimeMs: 1584450847106
> (org.apache.kafka.common.utils.AppInfoParser)
> [2020-03-17 13:14:07,204] INFO [Producer clientId=producer-11] Cluster ID:
> wif2mnkZTayzAEOb2VyoLA (org.apache.kafka.clients.Metadata)
>
> It seems that MM2 restarts the connectors in 2 seconds.
> But then I see the usual logs, but according to MM2's metrics, it is not
> mirroring any messages for about 5 minutes.
>
> [2020-03-17 13:14:10,485] INFO
> WorkerSourceTask{id=MirrorCheckpointConnector-3} Committing offsets
> (org.apache.kafka.connect.runtime.WorkerSourceTask)
> [2020-03-17 13:14:10,485] INFO
> WorkerSourceTask{id=MirrorCheckpointConnector-3} Committing offsets
> (org.apache.kafka.connect.runtime.WorkerSourceTask)
> [2020-03-17 13:14:10,485] INFO
> WorkerSourceTask{id=MirrorCheckpointConnector-3} flushing 0 outstanding
> messages for offset commit
> (org.apache.kafka.connect.runtime.WorkerSourceTask)
> [2020-03-17 13:14:10,485] INFO
> WorkerSourceTask{id=MirrorCheckpointConnector-3} flushing 0 outstanding
> messages for offset commit
> (org.apache.kafka.connect.runtime.WorkerSourceTask)
> [2020-03-17 13:14:10,637] INFO
> WorkerSourceTask{id=MirrorCheckpointConnector-3} Finished commitOffsets
> successfully in 152 ms (org.apache.kafka.connect.runtime.WorkerSourceTask)
> [2020-03-17 13:14:10,637] INFO
> WorkerSourceTask{id=MirrorCheckpointConnector-3} Finished commitOffsets
> successfully in 152 ms (org.apache.kafka.connect.runtime.WorkerSourceTask)
>
> So after ~5 minutes, I see that it is subscribing to the topics... and I
> also see in the metrics that that is the time when it starts to mirror
> messages.
>
> [2020-03-17 13:19:39,850] INFO [Consumer clientId=consumer-39,
> groupId=null] Subscribed to partition(s): ...
> [2020-03-17 13:19:39,851] INFO Starting with 308 previously uncommitted
> partitions. (org.apache.kafka.connect.mirror.MirrorSourceTask)
> [2020-03-17 13:19:39,851] INFO [Consumer clientId=consumer-39,
> groupId=null] Seeking to offset 178508243 for partition
> twTasks.austrac-service.executeTask.default-0 (org.apache.kafka
>
>
> Is there any idea how to speed this up?
>
> Thanks,
> Peter
>
