Hi Ashok, In general, what Ryanne said is correct. For example, if you send an email, and the send times out (but the email actually did get sent), your app cannot distinguish this from a failure in which the send times out before the email makes it out. Therefore, your only option would be to retry, which may result in duplicate messages. This obviously has nothing to do with Kafka, it's just a fact of life with distributed systems.
However, there's some possibility that you can use Kafka's guarantees to reduce your exposure to duplicate emails... But it's hard to say without more information about the structure of your application. Feel free to provide more info, and we'll see if we can give you some advice. Thanks, -John On Fri, May 10, 2019 at 10:11 AM Ryanne Dolan <ryannedo...@gmail.com> wrote: > > Kafka only supports exactly-once and idempotency within the context of > streams apps where records are consumed and produced within the same > cluster. As soon as you touch the outside world in a non-idempotent way, > e.g. by sending an email, these guarantees fall away. It is essentially > impossible to guarantee that an intended action occurs no more than once > while also guaranteeing that no intended action is skipped. This is a > problem with distributed systems in general. > > You might use something like a cache to prevent most spurious emails -- if > the cache already has record of the email being sent, don't resend it -- > but this will not prevent all duplicates. > > Ryanne > > On Fri, May 10, 2019 at 7:26 AM ASHOK MACHERLA <iash...@outlook.com> wrote: > > > *Dear Team* > > > > > > > > In our project, for SMS/Email purpose we are using Kafka cluster and > > Real-time Notification which is our custom application. > > > > > > > > We are sending to messages from *Kafka to Real-time Notification, and > > then SMTP Gateway* servers. > > > > > > > > Our problem is ,sometimes customers are getting same email for multiple > > times. > > > > During this time consumer is goes to Rebalancing mode. > > > > > > > > How to overcome this, > > > > Right now, we have 10 partitions for Kafka topic and 10 consumers. > > > > > > > > Can you please suggest us to fix this one. > > > > > > > > If you required any information/logs, I’ll share to you . > > > > > > > > Please help us, Thanks > > > > > > > > > > > > Sent from Mail <https://go.microsoft.com/fwlink/?LinkId=550986> for > > Windows 10 > > > > > >
