[
https://issues.apache.org/jira/browse/KAFKA-4040?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15421243#comment-15421243
]
Mathias Herberts commented on KAFKA-4040:
-----------------------------------------
The drawback could be overcome by sorting the consumers in a different way,
right now they are sorted lexicographically meaning that they cluster per
group.id then host, so all the consumers for a given groupid/host are assigned
partitions first. Maybe actually shuffling the consumers would lead to a better
balanced assignment.
The Fair Assignment Strategy seems to want to overcome the strictness of the
Round Robin, I'm more concerned with the level at which the round robin is
actually done, I would want to see it done at the host level, not at the thread
level.
> Round Robin Assignment does not create balanced assignments
> -----------------------------------------------------------
>
> Key: KAFKA-4040
> URL: https://issues.apache.org/jira/browse/KAFKA-4040
> Project: Kafka
> Issue Type: Improvement
> Affects Versions: 0.8.2.2
> Reporter: Mathias Herberts
>
> The RoundRobinAssignor loops over the consumers in a circular manner,
> assigning each time N partitions to each consumer, where N is the number of
> consuming threads.
> In the following scenario, this creates a major imbalance in the assignment.
> single topic with 64 partitions, 4 consumers (A,B,C,D), each with 12 threads.
> The roundrobin strategy will allocate the partitions in the following manner:
> A: 24
> B: 16
> C: 12
> D: 12
> when the expected assignment would rather be 16 to each consumer.
> The reason for this imbalance is that instead of allocating a single
> partition to each consumer in sequence, the assignor attempts to assign 12
> partitions each time it is handed a consumer by the circularIterator, so it
> starts by assigning 12 to each of A,B,C and D, then it has 16 partitions left
> and it is handed A again, to which it assigns 12 partitions before moving to
> B to which 4 will be assigned.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
