Hi Praveen, I don't know if you that's your case but if you know that consumers will lost ownership of partitions, you have to use the *ConsumerRebalanceListener* to drop the last offset of record processed in a clean way.
If you don't do that, the rebalance will start when the GroupCoordinator stopped to receive heartbeats from the Consumer. I don't know if that will solve your problem, but the consumer after commiting his last offset will leave the ConsumerGroup. Regards On Thu, Dec 7, 2017 at 9:58 PM Praveen <praveev...@gmail.com> wrote: > I have 4 consumers on 2 boxes (running two consumers each) and 16 > partitions. Each consumer takes 4 partitions. > > In Kafka 0.9.0.1, I'm noticing that even when a consumer is no longer > assigned the partition, it is able to commit offset to it. > > *Box 1 Started* > t1 - Box 1, Consumer 1 - Owns 8 partitions > Box 1, Consumer 2 - Owns 8 partitions > > Consumers start polling and are submitting tasks to a task pool for > processing. > > *Box 2 Started* > t2 - Box 1, Consumer 1 - Owns 4 partitions > Box 1, Consumer 2 - Owns 4 partitions > Box 2, Consumer 1 - Owns 4 partitions > Box 2, Consumer 2 - Owns 4 partitions > > Partition-1 is now reassigned to Box 2, Consumer 1. > But Box 1, Consumer 1 already submitted some of the records for > processing when it owned the partition earlier. > > t3 - Box 1, Consumer 1 - After the tasks finish executing, even tho it > longer owns the partition, it is still able to commit the offset > > t4 - Box 2, Consumer 1 - Commits offsets as well, overwriting offset > committed by Box 1, Consumer 1. > > Is this expected? Should I be using the ConsumerRebalanceListener to > prevent commits to partitions not owned by the consumer? > > - Praveen > -- Saïd BOURAS Consultant Big Data Mobile: 0662988731 Zenika Paris 10 rue de Milan 75009 Paris Standard : +33(0)1 45 26 19 15 <+33(0)145261915> - Fax : +33(0)1 72 70 45 10 <+33(0)172704510>
