Hi Vangelis, without transaction, Kafka (including MM) is either "at-least" 
once or "at-most" once. Most use cases prefer "at-least" once

https://supergloo.com/kafka/kafka-architecture-delivery/#:~:text=Now%2C%20Kafka%20provides%20%E2%80%9Cat%2D,case%2C%20data%20could%20be%20duplicated.&text=Processing%20in%20batches%20of%20records%20is%20available%20in%20Kafka%20as%20well.

On 2021/03/18 08:34:39, Vangelis Typaldos <vty...@m45.gr> wrote: 
> Thank you Ning for your response.
> 
> Could you please indicate if there is some statement or specific code part 
> where is mentioned the “at-least” delivery guarantee? Just for the record.
> 
> Kind Regards,
> 
> Από: Ning Zhang
> Αποστολή: Τετάρτη, 17 Μαρτίου 2021 22:39
> Προς: users@kafka.apache.org
> Θέμα: Re: Mirrormaker 2.0 - duplicates with idempotence enabled
> 
> Hello Vangelis,
> 
> By default, current MM 2.0 guarantees "at-least" once delivery guarantee, 
> meaning there will be duplicate messages under some failure scenarios.
> 
> If you prefer to no-message loss, there is a pending PR about MM 2.0
> 
> https://issues.apache.org/jira/browse/KAFKA-10339
> 
> On 2021/03/10 07:45:17, Vangelis Typaldos <vty...@m45.gr> wrote: 
> > Hi,
> > 
> > I have setup mirrormaker2 (Kafka v.2.6.0) on 2 clusters (CL1,CL2) and the 
> > mirroring seems to work properly except with an issue with duplicates in 
> > the following scenario:
> > While both clusters are up and running i simulate an incident, stopping one 
> > by one the brokers of the CL2 cluster. Stopping the first two brokers does 
> > not generate any issue. All messages of my test topic are mirrored without 
> > problems on CL1.topic of CL2 cluster. After stopping the last broker, 
> > obviously will stop mirroring messages in the CL2 side as all brokers are 
> > down. There is always active a producer that feeds with messages during the 
> > test on topic of CL1.
> > The problem starts on restarting the brokers. After starting the first 
> > broker i note that some messages (about 5%) are duplicated. I have 
> > connected a client on CL1.topic and i can confirm that indeed there are 
> > duplicated messages in my mirrored topic. 
> > Kindly suggest how i could avoid these duplicates. Idempotence may not work 
> > correctly during broker shutdown?
> > In the following you can find my MM2 relative config
> > clusters = CL1, CL2
> > CL1.bootstrap.servers = broker1CL1:9092, broker2CL1:9092, broker3CL1:9092
> > CL2.bootstrap.servers = broker1CL2:9092, broker2CL2:9092, broker3CL2:9092
> > 
> > PRIM->DSTR.enabled = true
> > DSTR->PRIM.enabled = true
> > 
> > CL1.producer.enable.idempotence = true
> > CL1.producer.acks=all
> > CL1.producer.max.in.flight.requests.per.connection=5
> > CL1.producer.retries=2147483647
> > CL1.consumer.isolation.level=read_committed
> > CL2.producer.enable.idempotence = true
> > CL2.producer.acks=all
> > CL2.producer.max.in.flight.requests.per.connection=5
> > CL2.producer.retries=2147483647
> > CL2.consumer.isolation.level=read_committed
> > 
> > 
> > Best Regards,
> > 
> 
> 

Reply via email to