I have 4 consumers on 2 boxes (running two consumers each) and 16 partitions. Each consumer takes 4 partitions.
In Kafka 0.9.0.1, I'm noticing that even when a consumer is no longer assigned the partition, it is able to commit offset to it. *Box 1 Started* t1 - Box 1, Consumer 1 - Owns 8 partitions Box 1, Consumer 2 - Owns 8 partitions Consumers start polling and are submitting tasks to a task pool for processing. *Box 2 Started* t2 - Box 1, Consumer 1 - Owns 4 partitions Box 1, Consumer 2 - Owns 4 partitions Box 2, Consumer 1 - Owns 4 partitions Box 2, Consumer 2 - Owns 4 partitions Partition-1 is now reassigned to Box 2, Consumer 1. But Box 1, Consumer 1 already submitted some of the records for processing when it owned the partition earlier. t3 - Box 1, Consumer 1 - After the tasks finish executing, even tho it longer owns the partition, it is still able to commit the offset t4 - Box 2, Consumer 1 - Commits offsets as well, overwriting offset committed by Box 1, Consumer 1. Is this expected? Should I be using the ConsumerRebalanceListener to prevent commits to partitions not owned by the consumer? - Praveen