Hey,
Running a MM2 cluster to mirror from A->B clusters I noticed that when I
add a new topic to A cluster, MM2 will notice it:
[2020-03-17 13:14:05,477] INFO Found 2719 topic-partitions on main. 1 are
new. 0 were removed. Previously had 2718.
(org.apache.kafka.connect.mirror.MirrorSourceConnector)
That's fine. It seems that MM2 connectors just simply restart to start up
with the new configuration.
My problem is that it takes about 5 minutes for MM2 to start to mirror
messages again.
What I see in the logs are:
[2020-03-17 13:14:07,107] INFO Kafka startTimeMs: 1584450847106
(org.apache.kafka.common.utils.AppInfoParser)
[2020-03-17 13:14:07,204] INFO [Producer clientId=producer-11] Cluster ID:
wif2mnkZTayzAEOb2VyoLA (org.apache.kafka.clients.Metadata)
It seems that MM2 restarts the connectors in 2 seconds.
But then I see the usual logs, but according to MM2's metrics, it is not
mirroring any messages for about 5 minutes.
[2020-03-17 13:14:10,485] INFO
WorkerSourceTask{id=MirrorCheckpointConnector-3} Committing offsets
(org.apache.kafka.connect.runtime.WorkerSourceTask)
[2020-03-17 13:14:10,485] INFO
WorkerSourceTask{id=MirrorCheckpointConnector-3} Committing offsets
(org.apache.kafka.connect.runtime.WorkerSourceTask)
[2020-03-17 13:14:10,485] INFO
WorkerSourceTask{id=MirrorCheckpointConnector-3} flushing 0 outstanding
messages for offset commit
(org.apache.kafka.connect.runtime.WorkerSourceTask)
[2020-03-17 13:14:10,485] INFO
WorkerSourceTask{id=MirrorCheckpointConnector-3} flushing 0 outstanding
messages for offset commit
(org.apache.kafka.connect.runtime.WorkerSourceTask)
[2020-03-17 13:14:10,637] INFO
WorkerSourceTask{id=MirrorCheckpointConnector-3} Finished commitOffsets
successfully in 152 ms (org.apache.kafka.connect.runtime.WorkerSourceTask)
[2020-03-17 13:14:10,637] INFO
WorkerSourceTask{id=MirrorCheckpointConnector-3} Finished commitOffsets
successfully in 152 ms (org.apache.kafka.connect.runtime.WorkerSourceTask)
So after ~5 minutes, I see that it is subscribing to the topics... and I
also see in the metrics that that is the time when it starts to mirror
messages.
[2020-03-17 13:19:39,850] INFO [Consumer clientId=consumer-39,
groupId=null] Subscribed to partition(s): ...
[2020-03-17 13:19:39,851] INFO Starting with 308 previously uncommitted
partitions. (org.apache.kafka.connect.mirror.MirrorSourceTask)
[2020-03-17 13:19:39,851] INFO [Consumer clientId=consumer-39,
groupId=null] Seeking to offset 178508243 for partition
twTasks.austrac-service.executeTask.default-0 (org.apache.kafka
Is there any idea how to speed this up?
Thanks,
Peter
