[
https://issues.apache.org/jira/browse/KAFKA-4677?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15888161#comment-15888161
]
ASF GitHub Bot commented on KAFKA-4677:
---------------------------------------
GitHub user dguy opened a pull request:
https://github.com/apache/kafka/pull/2609
KAFKA-4677: [Follow Up] add optimization to StickyTaskAssignor for rolling
rebounce
Detect when a rebalance has happened due to one or more existing nodes
bouncing. Keep assignment of previous active tasks the same and only assign the
tasks that were not active to the new clients.
You can merge this pull request into a Git repository by running:
$ git pull https://github.com/dguy/kafka kstreams-575
Alternatively you can review and apply these changes as the patch at:
https://github.com/apache/kafka/pull/2609.patch
To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:
This closes #2609
----
----
> Avoid unnecessary task movement across threads during rebalance
> ---------------------------------------------------------------
>
> Key: KAFKA-4677
> URL: https://issues.apache.org/jira/browse/KAFKA-4677
> Project: Kafka
> Issue Type: Bug
> Components: streams
> Affects Versions: 0.10.2.0
> Reporter: Damian Guy
> Assignee: Damian Guy
> Fix For: 0.10.3.0
>
>
> StreamPartitionAssigner tries to follow a sticky assignment policy to avoid
> expensive task migration. Currently, it does this in a best-effort approach.
> We could observe a case, for which tasks did migrate for no good reason, thus
> we assume that the current implementation could be improved to be more sticky.
> The concrete scenario is as follows:
> assume we have topology with 3 tasks, A, B, C
> assume we have 3 threads, each executing one task: 1-A, 2-B, 3-C
> for some reason, thread 1 goes down and a rebalance gets triggered
> thread 2 and 3 get their partitions revoked
> sometimes (not sure what the exact condition for this is), the new assignment
> flips the assignment for task B and C (task A is newly assigned to either
> thread 2 or 3)
> > possible new assignment 2(A,C) and 3-B
> There is no obvious reason (like load-balancing) why the task assignment for
> B and C does change to the other thread resulting in unnecessary task
> migration.
--
This message was sent by Atlassian JIRA
(v6.3.15#6346)
