Ah, I'm sorry, this is not the setting you need (it's actually the period
before "Found ... topic-partitions").
Please, ignore my message :)
Ivan
On Tue, 17 Mar 2020 at 18:04, Ivan Yurchenko <ivan0yurche...@gmail.com>
wrote:
> Hi,
>
> You need to set refresh.topics.interval.seconds, which is 10 minutes by
> default.
>
> Ivan
>
>
> On Tue, 17 Mar 2020 at 17:45, Péter Sinóros-Szabó
> <peter.sinoros-sz...@transferwise.com.invalid> wrote:
>
>> Hey,
>>
>> Running a MM2 cluster to mirror from A->B clusters I noticed that when I
>> add a new topic to A cluster, MM2 will notice it:
>> [2020-03-17 13:14:05,477] INFO Found 2719 topic-partitions on main. 1 are
>> new. 0 were removed. Previously had 2718.
>> (org.apache.kafka.connect.mirror.MirrorSourceConnector)
>>
>> That's fine. It seems that MM2 connectors just simply restart to start up
>> with the new configuration.
>> My problem is that it takes about 5 minutes for MM2 to start to mirror
>> messages again.
>>
>> What I see in the logs are:
>> [2020-03-17 13:14:07,107] INFO Kafka startTimeMs: 1584450847106
>> (org.apache.kafka.common.utils.AppInfoParser)
>> [2020-03-17 13:14:07,204] INFO [Producer clientId=producer-11] Cluster ID:
>> wif2mnkZTayzAEOb2VyoLA (org.apache.kafka.clients.Metadata)
>>
>> It seems that MM2 restarts the connectors in 2 seconds.
>> But then I see the usual logs, but according to MM2's metrics, it is not
>> mirroring any messages for about 5 minutes.
>>
>> [2020-03-17 13:14:10,485] INFO
>> WorkerSourceTask{id=MirrorCheckpointConnector-3} Committing offsets
>> (org.apache.kafka.connect.runtime.WorkerSourceTask)
>> [2020-03-17 13:14:10,485] INFO
>> WorkerSourceTask{id=MirrorCheckpointConnector-3} Committing offsets
>> (org.apache.kafka.connect.runtime.WorkerSourceTask)
>> [2020-03-17 13:14:10,485] INFO
>> WorkerSourceTask{id=MirrorCheckpointConnector-3} flushing 0 outstanding
>> messages for offset commit
>> (org.apache.kafka.connect.runtime.WorkerSourceTask)
>> [2020-03-17 13:14:10,485] INFO
>> WorkerSourceTask{id=MirrorCheckpointConnector-3} flushing 0 outstanding
>> messages for offset commit
>> (org.apache.kafka.connect.runtime.WorkerSourceTask)
>> [2020-03-17 13:14:10,637] INFO
>> WorkerSourceTask{id=MirrorCheckpointConnector-3} Finished commitOffsets
>> successfully in 152 ms (org.apache.kafka.connect.runtime.WorkerSourceTask)
>> [2020-03-17 13:14:10,637] INFO
>> WorkerSourceTask{id=MirrorCheckpointConnector-3} Finished commitOffsets
>> successfully in 152 ms (org.apache.kafka.connect.runtime.WorkerSourceTask)
>>
>> So after ~5 minutes, I see that it is subscribing to the topics... and I
>> also see in the metrics that that is the time when it starts to mirror
>> messages.
>>
>> [2020-03-17 13:19:39,850] INFO [Consumer clientId=consumer-39,
>> groupId=null] Subscribed to partition(s): ...
>> [2020-03-17 13:19:39,851] INFO Starting with 308 previously uncommitted
>> partitions. (org.apache.kafka.connect.mirror.MirrorSourceTask)
>> [2020-03-17 13:19:39,851] INFO [Consumer clientId=consumer-39,
>> groupId=null] Seeking to offset 178508243 for partition
>> twTasks.austrac-service.executeTask.default-0 (org.apache.kafka
>>
>>
>> Is there any idea how to speed this up?
>>
>> Thanks,
>> Peter
>>
>
