It should just continue consuming using the existing offsets. It'll have to refresh metadata to pick up the leadership change, but once it does it can just pick up where consumption from the previous leader stopped -- all the ISRs should have the same data, so the new leader will have all the same data the previous leader had (assuming unclean leader election is not enabled).
On Tue, Jul 21, 2015 at 9:11 PM, James Cheng <jch...@tivo.com> wrote: > > > On Jul 21, 2015, at 9:15 AM, Ewen Cheslack-Postava <e...@confluent.io> > wrote: > > > > On Tue, Jul 21, 2015 at 2:38 AM, Stevo Slavić <ssla...@gmail.com> wrote: > > > >> Hello Apache Kafka community, > >> > >> I find new consumer poll/seek javadoc a bit confusing. Just by reading > docs > >> I'm not sure what the outcome will be, what is expected in following > >> scenario: > >> > >> - kafkaConsumer is instantiated with auto-commit off > >> - kafkaConsumer.subscribe(someTopic) > >> - kafkaConsumer.position is called for every TopicPartition HLC is > actively > >> subscribed on > >> > >> and then when doing multiple poll calls in succession (without calling > >> commit), does seek have to be called in between poll calls to position > HLC > >> to skip what was read in previous poll, or does HLC keep that state > >> (position after poll) in memory, so that next poll (without seek in > between > >> two poll calls) will continue from where last poll stopped? > >> > > > > The position is tracked in-memory within the consumer, so as long as > there > > isn't a consumer rebalance, consumption will just proceed with subsequent > > messages (i.e. the behavior I think most people would find intuitive). > > However, if a rebalance occurs (another consumer instance joins the group > > or some leave), then a partition may be assigned to an different consumer > > instance that has no idea about the current position and will restart > based > > on the offset reset setting (because attempting to fetch the committed > > offset will fail since no offsets have been committed). > > > > Ewen, > > What happens if there is a broker failure and a new broker becomes the > partition leader? Does the high level consumer start listening to the new > partition leader at the in-memory position, or does it restart based on > saved offsets? > > Thanks, > -James > > > -Ewen > > > > > >> Could be it's just me not understanding this from javadoc. If not, maybe > >> javadoc can be improved to make this (even) more obvious. > >> > >> Kind regards, > >> Stevo Slavic. > >> > > > > > > > > -- > > Thanks, > > Ewen > > -- Thanks, Ewen
