Amir, We had similar requirement to consume every message reliably; the approach I picked was to push any message with unsuccessful consumption to a secondary topic for later entertainment; in our case the message/events were non-dependant so we use to make second attempt for consumption and on any failure push it to poison queue (Elastics Search in our case).
Regards, ajay On 8/16/16, 1:50 AM, "Amir Zuker" <a.zu...@gmail.com> wrote: >Hi everyone, > >I have a question regarding the 'KafkaConsumer' and its API in regards to >committing offsets. (kafka-clients 0.9.0.1) > >*The scenario -* >I am working with auto commit set to disabled because I want to implement >a >retry mechanism and eventually transfer the message to another topic that >contains the poison messages. >Since I want it to be reliable, I am not using the auto commit and I wish >to take control on when that should happen > >*The implementation detail -* >My class that extends 'Runnable' and is created by the KafkaConsumer needs >to commit the offset once it is done with handling the topic message. >However, the API for committing messages is located on the KafkaConsumer >with no relation to partition or thread. > >*The problem -* >If I understand correctly, I can use the same KafkaConsumer instance with >multiple threads against multiple partitions. >If that is the case, how can I commit the offset specific to my 'Runnable' >instance that just processed a single message without affecting other >threads and partitions? > >Thanks in advance, >Amir Zuker
