Re: Kafka connect replication using MirrorMaker 2.0

2021-03-22 Thread Ning Zhang
Hi Daniel, it is probably hard to figure out how to sync topics across two kafka connect cluster. In general, if implementing a solution requires strong technical pre-requisite (e.g. kafka connect will be aware of each other offset), It may be better to go with simpler solution first, for examp

Re: Kafka connect replication using MirrorMaker 2.0

2021-03-19 Thread dandaniel97
Hello and thank you for the reply! My problem is not with consumption of messages, because as you said, MirrorMaker2 knows how to deal with the consumer offsets. Rather my problem is with source connectors and the topic connect-offsets. Because Kafka connect manages where it stopped reading f

Re: Kafka connect replication using MirrorMaker 2.0

2021-03-19 Thread Ning Zhang
Hi Daniel, MirrorMaker2 creates its own "offsets" topic to track the process of consumption. just my 2 cents - If you already have two Kafka connect clusters in two different sites, it sounds practical to: (1) use "cluster" mode, instead of "dedicated" mode of MirrorMaker2 (2) add one "MirrorMak

Kafka connect replication using MirrorMaker 2.0

2021-03-18 Thread Daniel Beilin
Hi everyone, I'm trying to create an active-active deployment of a kafka cluster between two data centers using MirrorMaker2, but I'm facing a problem. In my deployment I have Kafka Connect in both sites which each of them connect to different database using sink and source connectors (MongoDB sour