So, when using the high level consumer, only offset.storage needs to be set, and the High level API will take care of committing the offsets automatically every so often.
If using the simple consumer API, call commitOffsets and set versionId on the OffsetCommitRequest¹s constructor to be 1 to commit the offsets to kafka. Also, I think that its best to avoid commit offsets after each message since it requires a network round trip. And this would really hurt throughput. -Erik On 9/19/15, 12:33 AM, "Andy Chambers" <achambers.h...@gmail.com> wrote: >Hi All, > >I'm a little bit confused when I read about offset management in kafka. A >google search turns up the wiki page that provides some example code about >how to use the offset API[1] but also buried in the docs is the >configuration option "offset.storage"[2]. > >So my question is "can we achieve scalable offset management just by >setting "offset.storage=kafka", and calling "consumer.commitOffsets" after >processing each message?" Or would we need to adapt the code in [1] to our >particular needs. > >Cheers, >Andy > >[1]: >https://cwiki.apache.org/confluence/display/KAFKA/Committing+and+fetching+ >consumer+offsets+in+Kafka >[2]: http://kafka.apache.org/documentation.html
