sanghyeok An created KAFKA-21169:
------------------------------------
Summary: Streams group coordinator may trigger unnecessary
rebalances on every heartbeat when a custom assignor returns empty task sets
Key: KAFKA-21169
URL: https://issues.apache.org/jira/browse/KAFKA-21169
Project: Kafka
Issue Type: Bug
Reporter: sanghyeok An
Assignee: sanghyeok An
Custom task assignors may use different representations for a subtopology with
no assigned tasks. Some implementations omit the subtopology entry, while
others include it with an empty task set:
{code:java}
// No standby tasks: omit the subtopology entry.
Map.of()
// No standby tasks: include the subtopology with an empty set.
Map.of("subtopology-1", Set.of()){code}
Both representations assign zero standby tasks. The public MemberAssignment API
does not explicitly require empty subtopology entries to be omitted, and the
coordinator accepts and persists them.
However, the target assignment preserves empty entries, while reconciliation
omits them from the current assignment. TasksTuple.sameTasks() treats these
representations as different. As a result, an otherwise stable Streams group
can trigger an unnecessary assignment refinement on every subsequent heartbeat,
even though its actual task assignment has not changed.
This advances the group, target assignment, and member epochs and generates
additional coordinator records. Reconciliation leaves the same representation
difference in place, so the next heartbeat triggers another update.
Here, an unnecessary rebalance means an epoch and assignment update in the
coordinator. In the reproduction, the assignor was called only once; the
subsequent updates did not require rerunning it. Task restarts or data loss
have not been demonstrated.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
