Thanks Jay, this is helpful information. Controlling when offsets are committed 
is possible only with the SimpleConsumer, correct? I believe that the 
ConsumerConnector (the high level consumer) commits the offsets automatically. 
It would be nice to have a hook into it that I can override to let it know when 
I finished processing a message so it can commit an offset. Is there an 
improvement request submitted for this already?

Thanks!

F.

On Jul 7, 2013, at 2:35 PM, Jay Kreps <jay.kr...@gmail.com> wrote:

> Consumer:
> The consumers position is controlled using a saved "offset" that marks its
> position in the topic/partition it is reading. This position is
> periodically updated. If you update the saved offset before processing
> messages you have the possibility of message loss if your consumer crashes
> before processing the messages. If you update the offset after processing
> the messages you have the possibility of duplicate messages when your
> consumer restarts as it will reprocess a few message it has already seen.
> You can control when the position is saved by calling commit().

Reply via email to