Luke Chen created KAFKA-12675:
---------------------------------
Summary: Improve sticky general assignor scalability and
performance
Key: KAFKA-12675
URL: https://issues.apache.org/jira/browse/KAFKA-12675
Project: Kafka
Issue Type: Improvement
Reporter: Luke Chen
Assignee: Luke Chen
Currently, we have "general assignor" for non-equal subscription case and
"constrained assignor" for all equal subscription case. There's a performance
test for constrained assignor with:
topicCount = {color:#0000ff}500{color};
partitionCount = {color:#0000ff}2000{color};
consumerCount = {color:#0000ff}2000{color};
in _testLargeAssignmentAndGroupWithUniformSubscription,_ total 1 million
partitions and we can complete the assignment within 2 second in my machine.
However, if we let 1 of the consumer subscribe to only 1 topic, it'll use
"general assignor", and the result with the same setting as above is:
*OutOfMemory,*
Even we down the count to:
topicCount = {color:#0000ff}50{color};
partitionCount = 1{color:#0000ff}000{color};
consumerCount = 1{color:#0000ff}000{color};
We still got *OutOfMemory*.
With this setting:
topicCount = {color:#0000ff}50{color};
partitionCount = 8{color:#0000ff}00{color};
consumerCount = 8{color:#0000ff}00{color};
We can complete with 10 seconds in my machine, which is still slow.
Since we are going to set default assignment strategy to
"CooperativeStickyAssignor" soon, we should improve the scalability and
performance for sticky general assignor.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
