Hello
I'm running a mirrormaker 2 cluster which copies from 3 source clusters
into 1 destination. Yesterday I restarted the cluster and it took 1 of the
mirrored topics a pretty long time to recover (2~ hours)
Since the restart the mm2 cluster has been sending a lot of these warning
messages from all 3 source clusters
WARN [Worker clientId=connect-4, groupId=local-ny-mm2] Catching up to
assignment's config offset.
(org.apache.kafka.connect.runtime.distributed.DistributedHerder:1020)
Here is a snippet of how the logs look like
TDOUT: [2020-07-29 05:44:17,143] INFO [Worker clientId=connect-2,
groupId=local-chi-mm2] Rebalance started
(org.apache.kafka.connect.runtime.distributed.WorkerCoordinator:222)
STDOUT: [2020-07-29 05:44:17,143] INFO [Worker clientId=connect-2,
groupId=local-chi-mm2] (Re-)joining group
(org.apache.kafka.clients.consumer.internals.AbstractCoordinator:552)
STDOUT: [2020-07-29 05:44:17,144] INFO [Worker clientId=connect-2,
groupId=local-chi-mm2] Successfully joined group with generation 9005
(org.apache.kafka.clients.consumer.internals.AbstractCoordinator:503)
STDOUT: [2020-07-29 05:44:17,144] INFO [Worker clientId=connect-2,
groupId=local-chi-mm2] Joined group at generation 9005 with protocol
version 2 and got assignment: Assignment{error=0,
leader='connect-1-cb7aa52c-a29a-4cf9-8f50-b691ba38aa3d',
leaderUrl='NOTUSED/local-chi', offset=1178, connectorIds=[], taskIds=[],
revokedConnectorIds=[], revokedTaskIds=[], delay=0} with rebalance delay: 0
(org.apache.kafka.connect.runtime.distributed.DistributedHerder:1549)
STDOUT: [2020-07-29 05:44:17,144] WARN [Worker clientId=connect-2,
groupId=local-chi-mm2] Catching up to assignment's config offset.
(org.apache.kafka.connect.runtime.distributed.DistributedHerder:1020)
STDOUT: [2020-07-29 05:44:17,144] INFO [Worker clientId=connect-2,
groupId=local-chi-mm2] Current config state offset -1 is behind group
assignment 1178, reading to end of config log
(org.apache.kafka.connect.runtime.distributed.DistributedHerder:1081)
STDOUT: [2020-07-29 05:44:17,183] INFO [Worker clientId=connect-6,
groupId=local-chi-mm2] Successfully joined group with generation 9317
(org.apache.kafka.clients.consumer.internals.AbstractCoordinator:503)
STDOUT: [2020-07-29 05:44:17,184] INFO [Worker clientId=connect-6,
groupId=local-chi-mm2] Joined group at generation 9317 with protocol
version 2 and got assignment: Assignment{error=0,
leader='connect-5-9a82cf55-e113-4112-86bf-d3cabcd44f54',
leaderUrl='NOTUSED/local-chi', offset=1401, connectorIds=[], taskIds=[],
revokedConnectorIds=[], revokedTaskIds=[], delay=0} with rebalance delay: 0
(org.apache.kafka.connect.runtime.distributed.DistributedHerder:1549)
STDOUT: [2020-07-29 05:44:17,184] WARN [Worker clientId=connect-6,
groupId=local-chi-mm2] Catching up to assignment's config offset.
(org.apache.kafka.connect.runtime.distributed.DistributedHerder:1020)
STDOUT: [2020-07-29 05:44:17,184] INFO [Worker clientId=connect-6,
groupId=local-chi-mm2] Current config state offset -1 is behind group
assignment 1401, reading to end of config log
(org.apache.kafka.connect.runtime.distributed.DistributedHerder:1081)
STDOUT: [2020-07-29 05:44:17,239] INFO [Worker clientId=connect-8,
groupId=local-sa-mm2] Finished reading to end of log and updated config
snapshot, new config log offset: -1
(org.apache.kafka.connect.runtime.distributed.DistributedHerder:1085)
STDOUT: [2020-07-29 05:44:17,239] INFO [Worker clientId=connect-8,
groupId=local-sa-mm2] Current config state offset -1 does not match group
assignment 1387. Forcing rebalance.
(org.apache.kafka.connect.runtime.distributed.DistributedHerder:1053)
STDOUT: [2020-07-29 05:44:17,239] INFO [Worker clientId=connect-8,
groupId=local-sa-mm2] Rebalance started
(org.apache.kafka.connect.runtime.distributed.WorkerCoordinator:222)
STDOUT: [2020-07-29 05:44:17,239] INFO [Worker clientId=connect-8,
groupId=local-sa-mm2] (Re-)joining group
(org.apache.kafka.clients.consumer.internals.AbstractCoordinator:552)
STDOUT: [2020-07-29 05:44:17,276] INFO [Worker clientId=connect-8,
groupId=local-sa-mm2] Successfully joined group with generation 9077
(org.apache.kafka.clients.consumer.internals.AbstractCoordinator:503)
Is anyone familiar with this and can help? So far didnt find anything
useful online specifically for mm2. I tried to delete the config topics for
each source cluster from the destination cluster and then restarting the
mm2 service again. This did seem to help somewhat but I am still getting
those warnings and more logs than expected per min.
Thanks,
Iftach
--
The above terms reflect a potential business arrangement, are provided
solely as a basis for further discussion, and are not intended to be and do
not constitute a legally binding obligation. No legally binding obligations
will be created, implied, or inferred until an agreement in final form is
executed in writing by all parties involved.
This email and any
attachments hereto may be confidential or privileged. If you received this
communication by mistake, please don't forward it to anyone else, please
erase all copies and attachments, and please let me know that it has gone
to the wrong person. Thanks.
