So, when using the high level consumer, only offset.storage needs to be
set, and the High level API will take care of committing the offsets
automatically every so often.

If using the simple consumer API, call commitOffsets and set versionId on
the OffsetCommitRequest¹s constructor to be 1 to commit the offsets to
kafka.

Also, I think that its best to avoid commit offsets after each message
since it requires a network round trip.  And this would really hurt
throughput.
-Erik    

On 9/19/15, 12:33 AM, "Andy Chambers" <achambers.h...@gmail.com> wrote:

>Hi All,
>
>I'm a little bit confused when I read about offset management in kafka. A
>google search turns up the wiki page that provides some example code about
>how to use the offset API[1] but also buried in the docs is the
>configuration option "offset.storage"[2].
>
>So my question is "can we achieve scalable offset management just by
>setting "offset.storage=kafka", and calling "consumer.commitOffsets" after
>processing each message?" Or would we need to adapt the code in [1] to our
>particular needs.
>
>Cheers,
>Andy
>
>[1]:
>https://cwiki.apache.org/confluence/display/KAFKA/Committing+and+fetching+
>consumer+offsets+in+Kafka
>[2]: http://kafka.apache.org/documentation.html

Reply via email to