I filed this jira, fwiw: https://issues.apache.org/jira/browse/KAFKA-2172
Jason
On Mon, Mar 23, 2015 at 2:44 PM, Jiangjie Qin
wrote:
> Hi Jason,
>
> Yes, I agree the restriction makes the usage of round-robin less flexible.
> I think the focus of round-robin strategy is workload balance. If
>
Hi Jason,
Yes, I agree the restriction makes the usage of round-robin less flexible.
I think the focus of round-robin strategy is workload balance. If
different consumers are consuming from different topics, it is unbalanced
by nature. In that case, is it possible that you use different consumer
g
Jiangjie,
Yeah, I welcome the round-robin strategy, as the 'range' strategy ('til now
the only one available), is not always good at balancing partitions, as you
observed above.
The main thing I'm bringing up in this thread though is the question of why
there needs to be a restriction to having a
I am not sure if that's how it works.
I suppose each consumer should be able to consume from all the topics right. If
not then it looks weird.
Thanks,
Mayuresh
Sent from my iPhone
> On Mar 20, 2015, at 7:15 PM, Jiangjie Qin wrote:
>
> Hi Jason,
>
> The motivation behind round robin is to
Hi Jason,
The motivation behind round robin is to better balance the consumers¹
load. Imagine you have two topics each with two partitions. These topics
are consumed by two consumers each with two consumer threads.
The range assignment gives:
T1-P1 -> C1-Thr1
T1-P2 -> C1-Thr2
T2-P1 -> C1-Thr1
T2
Jiangle,
The error messages I got (and the config doc) do clearly state that the
number of threads per consumer must match also
I'm not convinced that an easy to understand algorithm would work fine with
a heterogeneous set of selected topics between consumers.
Jason
On Thu, Mar 19, 2015 at
Hi Becket,
Can you list down an example for this. It would be easier to understand :)
Thanks,
Mayuresh
On Thu, Mar 19, 2015 at 4:46 PM, Jiangjie Qin
wrote:
> Hi Jason,
>
> The round-robin strategy first takes the partitions of all the topics a
> consumer is consuming from, then distributed th
Hi Jason,
The round-robin strategy first takes the partitions of all the topics a
consumer is consuming from, then distributed them across all the consumers.
If different consumers are consuming from different topics, the assigning
algorithm will generate different answers on different consumers.
So,
I've run into an issue migrating a consumer to use the new 'roundrobin'
partition.assignment.strategy. It turns out that several of our consumers
use the same group id, but instantiate several different consumer instances
(with different topic selectors and thread counts). Often, this is don
