Hi Raj, The Kafka design is based on fully decoupled producers and consumers, where the responsibility of the producer ends after a record has been successfully produced. Since a producer is not aware of other applications that might be consuming the record it can continue with its own flow. The same is true for a consumer, they are usually not aware which application is producing a record. only where to read a record and how to deserialize and interpret the record. This allows for new applications to be added as producers or consumers of the data.
This is one of the reasons why Kafka does not contain a builtin feature to send an acknowledgement to the producer when the record has been consumed. It has only built-in support to acknowledge delivery of a record to the partition, which is controlled by the acks setting in a producer. If the design requires an acknowledgement that the record is picked up by a consumer then the easiest way I can think of is to create a separate topic where the producer will be listening for acknowledgement. The flow would look something like this for the producing application 1. The producer gets a message to send to a topic. 2. The producer creates a record with a unique message id in the Kafka Header 3. The producer part sends the record to the topic, and adds the message id to a list of messages waiting for acknowledgement 4. The consumer part of the application polls for new records on the acknowledgement topic 5. The message id is cleared for from the waiting list, and additional actions can be taken according to the needs of your application The flow of the consuming application would look like this 1. The consumer polls for records on a specific topic. 2. When a record is received and correctly deserialized the message id is extracted 3. The message id is produced to the acknowledgement topic 4. The record is processed by the consuming application As you can see you're basically recreating a classical queueing mechanism with a feedback channel, requiring producing and consuming applications to include a lot of additional logic. Often you can solve this by redesigning the solution to make sure that the producer responsibilities end after getting the acknowledgement from the broker that the record was produced. The data is now available to any consumers, but it is up to the consumers responsibility to keep up with the records on the topic. This will take of course some time, and your stakeholders and fellow developers might need to be convinced of this new way of working. My apologies for the long explanation, but I hope you can find some use for this. Kind regards, Richard Bosch Developer Advocate Axual BV https://axual.com/ On Thu, Jul 28, 2022 at 4:09 PM selva raj selvaraj <selva_...@tecnics.com> wrote: > Hi Team! > > I developed a simple kafka producer & consumer module. Just I want to > acknowledge whether the message is delivered or not. I couldn't find any > document, reference or sample code. Could you please guide me on this? > > > *Development Language : *Python > > > Thanks! > Raj >
