> On May 18, 2015, 9:46 p.m., Guozhang Wang wrote: > > core/src/main/scala/kafka/coordinator/PartitionAssignor.scala, lines 87-89 > > <https://reviews.apache.org/r/34273/diff/1/?file=961395#file961395line87> > > > > I feel this logic's irregular assignment result may make it a bit > > harder to reason for debugging. For example if you have three consumers > > (1,2,3) with two topics each with three partitions, and consumer 1/2 > > subscribe to topic 1/2 while consumer 3 only subscribe to 2. The result > > would be: > > > > t1p0: c1 > > t1p1: c2 > > t1p2: c1 > > t2p0: c2 > > t2p0: c1 > > t2p0: c3 > > > > Instead it may be clearer to have: > > > > t1p0: c1 > > t1p1: c2 > > t1p2: c1 > > t2p0: c1 > > t2p1: c2 > > t2p2: c3 > > > > So I suggest grouping the consumers by subscribed topics, and for each > > group do the round-robin. > > Onur Karaman wrote: > If I understood you correctly, you're suggesting we group consumers on a > per-topic basis and reset your roundrobin consumer assignor position every > topic. > > This would basically become range. The specific partitions being assigned > would differ from range, but the ending quantity per consumer per topic would > be the same. This also means the left-most consumers are going to get the > most partitions overall. As an extreme example, imagine a group of 10 > consumers subscribed to 10 topics, each topic with 1 partition. Reseting the > consumer assignor means the leftmost consumer will get all 10 partitions. > > I think reseting the consumer assignor will in general bias assignments > towards the left. > > Were you referring to a non-per-topic grouping?
I was proposing to do round-robin for each subscription, like for your example there will be one subscription topic1-10 for consumers1-10 so each partition still get a different consumer, but I got your point that it will end up having per-topic grouping with overlapping subscriptions. - Guozhang ----------------------------------------------------------- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/34273/#review84223 ----------------------------------------------------------- On May 15, 2015, 3:46 p.m., Onur Karaman wrote: > > ----------------------------------------------------------- > This is an automatically generated e-mail. To reply, visit: > https://reviews.apache.org/r/34273/ > ----------------------------------------------------------- > > (Updated May 15, 2015, 3:46 p.m.) > > > Review request for kafka. > > > Bugs: KAFKA-2196 > https://issues.apache.org/jira/browse/KAFKA-2196 > > > Repository: kafka > > > Description > ------- > > roundrobin doesn't need to make all consumers have identical topic > subscriptions. > > todo: > - run this and range through some simulations > > > Diffs > ----- > > core/src/main/scala/kafka/coordinator/PartitionAssignor.scala > 106982286ce7a9e4f0e9722da2812e3a8e7a6cc3 > core/src/test/scala/unit/kafka/coordinator/PartitionAssignorTest.scala > ba6d5cd85b89214247209d974701eb6c9eb1e2b2 > > Diff: https://reviews.apache.org/r/34273/diff/ > > > Testing > ------- > > > Thanks, > > Onur Karaman > >
