Re: MM2 Question

2024-01-08 Thread Vinay Bagare
Thanks Greg! I am still gathering information but our internal team has confirmed that we have a separate cluster. I will pass along these details for them to investigate. Best, Vinay Bagare From: Greg Harris Date: Monday, January 8, 2024 at 5:40 PM To: users@kafka.apache.org Subject: Re: MM2

Re: MM2 Question

2024-01-08 Thread Greg Harris
Hi Vinay, I am sorry to hear about your difficulties with MirrorMaker2. Are you using the MirrorMaker2 dedicated mode, or do you have the MirrorMaker2 connectors running in a separate Connect cluster? If you're using the dedicated mode, that is a known problem on <3.5 and is the motivation for ht

MM2 Question

2024-01-08 Thread Vinay Bagare
Hi Team, I am fairly new to Kafka but one issue that we are currently battling is in MM2. MM2 for no reason is missing new topics that are present in one of the source clusters (no obvious error message in MM2 error logs). I heard from our internal team that this MM2 has 10k topics to take care

Re: Mirror Maker bidirectional offset sync

2024-01-08 Thread Ryanne Dolan
Jeroen, MirrorClient will correctly translate offsets for both failover and failback, exactly as you describe. It's possible to automate failover and failback using that logic. The integration tests automatically fail over and fail back, for example. I've seen it done two ways: during startup withi

Re: Mirror Maker bidirectional offset sync

2024-01-08 Thread Greg Harris
Hi Jeroen, Thanks for looking into MM2 for your use-case! I believe the "active-active" model that MM2 uses is really more of a pair of opposing "active-passive" replication flows, and this extends to the offset sync mechanism too. MM2 doesn't change the state of the source topic or consumer grou

Mirror Maker bidirectional offset sync

2024-01-08 Thread Jeroen Schutrup
Hi all, I'm exploring using the MirrorSourceConnector and MirrorCheckpointConnector on Kafka Connect to setup active/active replication between two Kafka clusters. Using the DefaultReplicationPolicy replication policy class, messages originating from the source cluster get replicated as expected to