Hi, I have setup mirrormaker2 (Kafka v.2.6.0) on 2 clusters (CL1,CL2) and the mirroring seems to work properly except with an issue with duplicates in the following scenario: While both clusters are up and running i simulate an incident, stopping one by one the brokers of the CL2 cluster. Stopping the first two brokers does not generate any issue. All messages of my test topic are mirrored without problems on CL1.topic of CL2 cluster. After stopping the last broker, obviously will stop mirroring messages in the CL2 side as all brokers are down. There is always active a producer that feeds with messages during the test on topic of CL1. The problem starts on restarting the brokers. After starting the first broker i note that some messages (about 5%) are duplicated. I have connected a client on CL1.topic and i can confirm that indeed there are duplicated messages in my mirrored topic. Kindly suggest how i could avoid these duplicates. Idempotence may not work correctly during broker shutdown? In the following you can find my MM2 relative config clusters = CL1, CL2 CL1.bootstrap.servers = broker1CL1:9092, broker2CL1:9092, broker3CL1:9092 CL2.bootstrap.servers = broker1CL2:9092, broker2CL2:9092, broker3CL2:9092
PRIM->DSTR.enabled = true DSTR->PRIM.enabled = true CL1.producer.enable.idempotence = true CL1.producer.acks=all CL1.producer.max.in.flight.requests.per.connection=5 CL1.producer.retries=2147483647 CL1.consumer.isolation.level=read_committed CL2.producer.enable.idempotence = true CL2.producer.acks=all CL2.producer.max.in.flight.requests.per.connection=5 CL2.producer.retries=2147483647 CL2.consumer.isolation.level=read_committed Best Regards,
