Hi Daniel, it is probably hard to figure out how to sync topics across two kafka connect cluster.
In general, if implementing a solution requires strong technical pre-requisite (e.g. kafka connect will be aware of each other offset), It may be better to go with simpler solution first, for example, "active-active" MM2 setup on one kafka connect cluster. On 2021/03/19 14:23:30, dandanie...@gmail.com wrote: > Hello and thank you for the reply! > > My problem is not with consumption of messages, because as you said, > MirrorMaker2 knows how to deal with the consumer offsets. Rather my problem > is with source connectors and the topic connect-offsets. > > Because Kafka connect manages where it stopped reading from a source using to > topic, the second connect cluster is not aware of the offsets and I can get > duplicates. I know MirrorMaker2 does not guarantee exactly-once delivery but > is there a way to maybe sync to topics so the kafka connect will be aware of > each other offset. > > Daniel > > > On 19 Mar 2021, at 9:09, Ning Zhang <ning2008w...@gmail.com> wrote: > > Hi Daniel, MirrorMaker2 creates its own "offsets" topic to track the > > process of consumption. > > > > just my 2 cents - If you already have two Kafka connect clusters in two > > different sites, it sounds practical to: > > (1) use "cluster" mode, instead of "dedicated" mode of MirrorMaker2 > > (2) add one "MirrorMaker" connector on each Kafka connect cluster and do > > "one-way" replication. As you have two "MirrorMaker" connectors, it should > > behave like "active-active" deployment. > > > > Just in case you only have one Kafka connect cluster, do "active-active" > > replication, rather than "one-way" > > > > On 2021/03/18 08:15:40, Daniel Beilin <dandanie...@gmail.com> wrote: > >> Hi everyone, > >> I'm trying to create an active-active deployment of a kafka cluster between > >> two data centers using MirrorMaker2, but I'm facing a problem. > >> In my deployment I have Kafka Connect in both sites which each of them > >> connect to different database using sink and source connectors (MongoDB > >> source connector , JDBC sink/source connector) > >> I’d like to know what’s the best practice for active-active is using Kafka > >> connect , since I noticed the “connect-offsets” topic is not replicated in > >> mm2. > >> > >> Best regards, > >> Daniel >
