Ravi Bhardwaj created KAFKA-12959:
-------------------------------------
Summary: Prioritize assigning standby tasks to threads without any
active tasks
Key: KAFKA-12959
URL: https://issues.apache.org/jira/browse/KAFKA-12959
Project: Kafka
Issue Type: Improvement
Components: streams
Affects Versions: 2.8.0
Reporter: Ravi Bhardwaj
Attachments: app1.log, app2.log
Currently while distributing the standby tasks streams does not check if there
are threads without any tasks or with less number of tasks. This can lead to
few threads getting assigned both active and standby tasks when are threads
within the same instance without any tasks assigned.
Example:
{code:java}
App 1:
[wordcount-lambda-example-client-StreamThread-1] Handle new assignment with:
New active tasks: [0_1]
New standby tasks: [1_0]
Existing active tasks: []
Existing standby tasks: [1_0]
[wordcount-lambda-example-client-StreamThread-2] Handle new assignment with:
New active tasks: [1_1]
New standby tasks: []
Existing active tasks: [1_1]
Existing standby tasks: []
[wordcount-lambda-example-client-StreamThread-3] Handle new assignment with:
New active tasks: []
New standby tasks: []
Existing active tasks: []
Existing standby tasks: []
[wordcount-lambda-example-client-StreamThread-4] Handle new assignment with:
New active tasks: []
New standby tasks: []
Existing active tasks: []
Existing standby tasks: []
{code}
{code:java}
App2:
[wordcount-lambda-example-client-StreamThread-1] Handle new assignment with:
New active tasks: [1_0]
New standby tasks: [1_1]
Existing active tasks: []
Existing standby tasks: [1_0, 1_1]
[wordcount-lambda-example-client-StreamThread-2] Handle new assignment with:
New active tasks: [0_0]
New standby tasks: []
Existing active tasks: []
Existing standby tasks: []
[wordcount-lambda-example-client-StreamThread-3] Handle new assignment with:
New active tasks: []
New standby tasks: []
Existing active tasks: []
Existing standby tasks: []
[wordcount-lambda-example-client-StreamThread-4] Handle new assignment with:
New active tasks: []
New standby tasks: []
Existing active tasks: []
Existing standby tasks: []
{code}
The standby tasks in both apps is assigned to Thread-1 even though it has an
active tasks when Thread-3 and Thread-4 didn't have any tasks assigned.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
