Hi,
I sent this as JIRA here (1) but no response so far.
I tried it with 0.8.3 instead of 0.8.1.1 and the behaviour seems to be the same.
I think that the main problem is that I am instantiating consumers
very quickly one after another in a loop an it seems that the
automatic rebalancing does not work properly in this case.
I tried to play with
partition.assignement.strategy both roundrobin and range
rebalance.max.retries
rebalance.backoff.ms
but non of these seems to work well.
My scenario:
5 producers, 1 topic with 10 partitions, on the consumer side I am
calling Consumers.createJavaConsumer() 10 time, each time with
topicMap("mytopic", 1).
The result is that I am consuming messages only but 4 or 5 consumers
instead of 10. I managed to have 7 consumers once and they were
assigned 3,2,1,1,1,1,1 partitions each.
So, messages are indeed fetched from all partitions and they are
processed correctly but I fail to understand why automatic rebalancing
does not assign partitions to remaining free consumers in case some
consumer has more then 1 partition assigned.
I really like the idea of automatic rebalancing but I want to commit
offsets per partition and this is only achieved by having 1 consumer
thread per consumer connector each thread holding 1 partition.
I would be even satisfied with committing to more then 1 partition in
case some consumer fails and its parititions are migrated to another
consumer however I can not even have all 10 consumers busy in the
first place.
Why partitions are not spread evenly accross all consumers?
(1) https://issues.apache.org/jira/browse/KAFKA-2331
--
Stefan Miklosovic
