Duplication can happen if your producer or consumer are exiting uncleanly. Like if producer just crashes before it receives ack from broker your logic will fail to register that message got produced. And when it comes back up it will try to send that batch again. Same with consumer, if it crashes before committing a batch of messages and comes back up it will receive that batch. Only help is to try exiting cleanly as far as possible. Ensure you catch kill signal then have your producer mark messages delivered to Kafka broker as processed. In case of consumer commit more often, catch kill signal and commit remaining. That's how I have done in my application.
On Mon, May 28, 2018 at 12:48 PM Karthick Kumar <kku...@apptivo.co.in> wrote: > Hi, > > Facing Duplication inconsistently while bouncing Kafka producer and > consumer in tomcat node. any help will be appreciated to find out the root > cause. > > -- > With Regards, > Karthick.K >
