Hello Vangelis, By default, current MM 2.0 guarantees "at-least" once delivery guarantee, meaning there will be duplicate messages under some failure scenarios.
If you prefer to no-message loss, there is a pending PR about MM 2.0 https://issues.apache.org/jira/browse/KAFKA-10339 On 2021/03/10 07:45:17, Vangelis Typaldos <vty...@m45.gr> wrote: > Hi, > > I have setup mirrormaker2 (Kafka v.2.6.0) on 2 clusters (CL1,CL2) and the > mirroring seems to work properly except with an issue with duplicates in the > following scenario: > While both clusters are up and running i simulate an incident, stopping one > by one the brokers of the CL2 cluster. Stopping the first two brokers does > not generate any issue. All messages of my test topic are mirrored without > problems on CL1.topic of CL2 cluster. After stopping the last broker, > obviously will stop mirroring messages in the CL2 side as all brokers are > down. There is always active a producer that feeds with messages during the > test on topic of CL1. > The problem starts on restarting the brokers. After starting the first broker > i note that some messages (about 5%) are duplicated. I have connected a > client on CL1.topic and i can confirm that indeed there are duplicated > messages in my mirrored topic. > Kindly suggest how i could avoid these duplicates. Idempotence may not work > correctly during broker shutdown? > In the following you can find my MM2 relative config > clusters = CL1, CL2 > CL1.bootstrap.servers = broker1CL1:9092, broker2CL1:9092, broker3CL1:9092 > CL2.bootstrap.servers = broker1CL2:9092, broker2CL2:9092, broker3CL2:9092 > > PRIM->DSTR.enabled = true > DSTR->PRIM.enabled = true > > CL1.producer.enable.idempotence = true > CL1.producer.acks=all > CL1.producer.max.in.flight.requests.per.connection=5 > CL1.producer.retries=2147483647 > CL1.consumer.isolation.level=read_committed > CL2.producer.enable.idempotence = true > CL2.producer.acks=all > CL2.producer.max.in.flight.requests.per.connection=5 > CL2.producer.retries=2147483647 > CL2.consumer.isolation.level=read_committed > > > Best Regards, >
