I have 4 consumers on 2 boxes (running two consumers each) and 16
partitions. Each consumer takes 4 partitions.
In Kafka 0.9.0.1, I'm noticing that even when a consumer is no longer
assigned the partition, it is able to commit offset to it.
*Box 1 Started*
t1 - Box 1, Consumer 1 - Owns 8 partitions
Box 1, Consumer 2 - Owns 8 partitions
Consumers start polling and are submitting tasks to a task pool for
processing.
*Box 2 Started*
t2 - Box 1, Consumer 1 - Owns 4 partitions
Box 1, Consumer 2 - Owns 4 partitions
Box 2, Consumer 1 - Owns 4 partitions
Box 2, Consumer 2 - Owns 4 partitions
Partition-1 is now reassigned to Box 2, Consumer 1.
But Box 1, Consumer 1 already submitted some of the records for
processing when it owned the partition earlier.
t3 - Box 1, Consumer 1 - After the tasks finish executing, even tho it
longer owns the partition, it is still able to commit the offset
t4 - Box 2, Consumer 1 - Commits offsets as well, overwriting offset
committed by Box 1, Consumer 1.
Is this expected? Should I be using the ConsumerRebalanceListener to
prevent commits to partitions not owned by the consumer?
- Praveen
