Re: On rebalance consumer group start from old offset

2017-07-20 Thread Justin Maltat
Solved by kafka-5600 Le mar. 18 juil. 2017 18:51, Sabarish Sasidharan a écrit : > This is similar to a problem I am also grappling with. We store the > processed offset for each partition in state store. And after restarts we > see that sometimes the start offset that Kafka Streams uses is a few

Re: On rebalance consumer group start from old offset

2017-07-18 Thread Sabarish Sasidharan
This is similar to a problem I am also grappling with. We store the processed offset for each partition in state store. And after restarts we see that sometimes the start offset that Kafka Streams uses is a few thousands to a couple million behind per partition. To compound it, this is not repeatab

On rebalance consumer group start from old offset

2017-07-18 Thread Justin Maltat
Hi On a 3 brokers cluster when one of the broker come back after a restart group rebalancing happens on our 2 consumers which make them restart to consume from an old offset which is not the earliest. Looking at the consumer offsets through kafka tools when running commits look good but on rebalan