Kafka only supports exactly-once and idempotency within the context of streams apps where records are consumed and produced within the same cluster. As soon as you touch the outside world in a non-idempotent way, e.g. by sending an email, these guarantees fall away. It is essentially impossible to guarantee that an intended action occurs no more than once while also guaranteeing that no intended action is skipped. This is a problem with distributed systems in general.
You might use something like a cache to prevent most spurious emails -- if the cache already has record of the email being sent, don't resend it -- but this will not prevent all duplicates. Ryanne On Fri, May 10, 2019 at 7:26 AM ASHOK MACHERLA <iash...@outlook.com> wrote: > *Dear Team* > > > > In our project, for SMS/Email purpose we are using Kafka cluster and > Real-time Notification which is our custom application. > > > > We are sending to messages from *Kafka to Real-time Notification, and > then SMTP Gateway* servers. > > > > Our problem is ,sometimes customers are getting same email for multiple > times. > > During this time consumer is goes to Rebalancing mode. > > > > How to overcome this, > > Right now, we have 10 partitions for Kafka topic and 10 consumers. > > > > Can you please suggest us to fix this one. > > > > If you required any information/logs, I’ll share to you . > > > > Please help us, Thanks > > > > > > Sent from Mail <https://go.microsoft.com/fwlink/?LinkId=550986> for > Windows 10 > > >
