Hi Vinay,

I am sorry to hear about your difficulties with MirrorMaker2. Are you
using the MirrorMaker2 dedicated mode, or do you have the MirrorMaker2
connectors running in a separate Connect cluster?

If you're using the dedicated mode, that is a known problem on <3.5
and is the motivation for
https://cwiki.apache.org/confluence/display/KAFKA/KIP-710%3A+Full+support+for+distributed+mode+in+dedicated+MirrorMaker+2.0+clusters
which was released in 3.5.0. See the KIP for how to enable the
feature.
The executive summary is: if MM2 does not start a REST server
(behavior <3.5 and default for >=3.5) then changes to the set of
topics cannot be communicated between nodes, so the task configs are
stuck with an outdated set of topics.
A workaround without upgrading or reconfiguring the cluster is to
fully stop and restart MirrorMaker2, to allow the first node joining
the cluster to refresh the set of topics and reconfigure the
replication tasks.

Otherwise, you may be interested to look for WARN logs, or particular
INFO logs such as this one [1].
If you're not using the dedicated mode, you can examine at the task
configs via the REST API [2] to see what the tasks are currently
configured to do.

[1]: 
https://github.com/apache/kafka/blob/a618f2095f2ce8bf345a8faa884c98ddc822c16b/connect/mirror/src/main/java/org/apache/kafka/connect/mirror/MirrorSourceConnector.java#L260-L265
[2]: https://kafka.apache.org/documentation/#connect_rest

I hope this helps in your investigation and remediation!

Thanks,
Greg

On Mon, Jan 8, 2024 at 3:03 PM Vinay Bagare <vinay.bag...@gmail.com> wrote:
>
> Hi Team,
>
> I am fairly new to Kafka but one issue that we are currently battling is in 
> MM2. MM2 for no reason is missing new topics that are present in one of the 
> source clusters (no obvious error message in MM2 error logs).
> I heard from our internal team that this MM2 has 10k topics to take care of 
> cluster of Brokers (Legacy design as per Application needs).
>
> Does anyone know if we are hitting any undocumented limits in MM2.
> I will try to get some additional information from our SRE team.
>
> Best,
> Vinay Bagare
>

Reply via email to