Kafka connect replication using MirrorMaker 2.0

2021-03-18 Thread Daniel Beilin
Hi everyone, I'm trying to create an active-active deployment of a kafka cluster between two data centers using MirrorMaker2, but I'm facing a problem. In my deployment I have Kafka Connect in both sites which each of them connect to different database using sink and source connectors (MongoDB sour

Re: Mirrormaker 2.0 - duplicates with idempotence enabled

2021-03-18 Thread Vangelis Typaldos
Thank you Ning for your response. Could you please indicate if there is some statement or specific code part where is mentioned the “at-least” delivery guarantee? Just for the record. Kind Regards, Από: Ning Zhang Αποστολή: Τετάρτη, 17 Μαρτίου 2021 22:39 Προς: users@kafka.apache.org Θέμα: Re: M

Re: Mirrormaker 2.0 - duplicates with idempotence enabled

2021-03-18 Thread Ning Zhang
Hi Vangelis, without transaction, Kafka (including MM) is either "at-least" once or "at-most" once. Most use cases prefer "at-least" once https://supergloo.com/kafka/kafka-architecture-delivery/#:~:text=Now%2C%20Kafka%20provides%20%E2%80%9Cat%2D,case%2C%20data%20could%20be%20duplicated.&text=Proc