ableegoldman commented on a change in pull request #8668:
URL: https://github.com/apache/kafka/pull/8668#discussion_r432802106
##########
File path:
clients/src/main/java/org/apache/kafka/clients/consumer/internals/AbstractStickyAssignor.java
##########
@@ -40,7 +43,11 @@
public static final int DEFAULT_GENERATION = -1;
- private PartitionMovements partitionMovements;
+ private PartitionMovements partitionMovements = new PartitionMovements();
+
+ // Keep track of the partitions being migrated from one consumer to
another during assignment
+ // so the cooperative assignor can adjust the assignment
+ protected Map<TopicPartition, String> partitionsTransferringOwnership =
new HashMap<>();
Review comment:
This is just an optimization for the cooperative case: I found that the
assignment time for the eager and cooperative assignor began to diverge once
you reached partition counts in the millions. At 10 million partitions for
example, the eager assignor hovered around 30s but the cooperative assignor was
upwards of 5-6 minutes.
The discrepancy was entirely due to the `adjustAssignment` method needing to
compute the set of partitions transferring ownership in the completed
assignment. But we can build up this map during assignment much more
efficiently, by taking advantage of the additional context we have at various
steps in the algorithm. Tracking and exposing this set to the cooperative
assignor cut the assignment time for large partition numbers pretty
drastically, putting the cooperative assignor on par with the eager assignor.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
