[
https://issues.apache.org/jira/browse/KAFKA-12675?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17324358#comment-17324358
]
Travis Bischel edited comment on KAFKA-12675 at 4/19/21, 11:58 PM:
-------------------------------------------------------------------
Great insight on getting rid of partition2AllPotentialConsumers, as well as
keeping some more things sorted! I was able to translate that into my own code
and dropped the large imbalance from 9.5s to 0.5s, as well as from 8.5G memory
util to 0.5G :)
I'll take a look at the code more in depth soon.
Edit: after further improvements I was able to get the large imbalance in my
client down to 220ms and 150MB.
was (Author: twmb):
Great insight on getting rid of partition2AllPotentialConsumers, as well as
keeping some more things sorted! I was able to translate that into my own code
and dropped the large imbalance from 9.5s to 0.5s, as well as from 8.5G memory
util to 0.5G :)
I'll take a look at the code more in depth soon.
Edit: after further improvements I was able to get the large imbalance down to
220ms and 171MB,
> Improve sticky general assignor scalability and performance
> -----------------------------------------------------------
>
> Key: KAFKA-12675
> URL: https://issues.apache.org/jira/browse/KAFKA-12675
> Project: Kafka
> Issue Type: Improvement
> Reporter: Luke Chen
> Assignee: Luke Chen
> Priority: Major
>
> Currently, we have "general assignor" for non-equal subscription case and
> "constrained assignor" for all equal subscription case. There's a performance
> test for constrained assignor with:
> topicCount = {color:#0000ff}500{color};
> partitionCount = {color:#0000ff}2000{color};
> consumerCount = {color:#0000ff}2000{color};
> in _testLargeAssignmentAndGroupWithUniformSubscription,_ total 1 million
> partitions and we can complete the assignment within 2 second in my machine.
> However, if we let 1 of the consumer subscribe to only 1 topic, it'll use
> "general assignor", and the result with the same setting as above is:
> *OutOfMemory,*
> Even we down the count to:
> topicCount = {color:#0000ff}50{color};
> partitionCount = 1{color:#0000ff}000{color};
> consumerCount = 1{color:#0000ff}000{color};
> We still got *OutOfMemory*.
> With this setting:
> topicCount = {color:#0000ff}50{color};
> partitionCount = 8{color:#0000ff}00{color};
> consumerCount = 8{color:#0000ff}00{color};
> We can complete in 10 seconds in my machine, which is still slow.
>
> Since we are going to set default assignment strategy to
> "CooperativeStickyAssignor" soon, we should improve the scalability and
> performance for sticky general assignor.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
