Matthias J. Sax created KAFKA-20998:
---------------------------------------
Summary: Add HA task assignor
Key: KAFKA-20998
URL: https://issues.apache.org/jira/browse/KAFKA-20998
Project: Kafka
Issue Type: Sub-task
Components: group-coordinator, streams
Reporter: Matthias J. Sax
The "classic" protocol offers two built-in assignors, "sticky" and
"ha_assignor". So far, "streams" only offers the sticky assignor.
KIP-1017 defines both "sticky" and "ha_assignor" as options, too, and we can
now close this gap.
However, we should amend the KIP slightly, and call the second built-in
assignor "balanced" (instead of "ha_assignor" as currently specified on the
KIP). The reason is, that high-availability features are delivered
independently on the assignment strategy with "streams" protocol, ie, "warmup
tasks" are inserted by the new refiner step, not the task assignor. Thus, the
new assignor is not a 1:1 of the classic HA-assignor, but only need to deliver
a different ("balanced") task assignment compare to sticky assignor.
We can start with a 1:1 translation of he existing task assignment algorithm,
but should use JMH benchmarks, and refine as needed, similar to what we did for
the sticky assignor. Because task assignment is running inside the GC, we need
to ensure that the task assignor performance is good, so it doesn't put too
much load on the broker.
The classic HA-assignor also supports rack-aware standby task assignment, which
we should exclude in this ticket. We rather want so deliver rack-aware standby
task assignment for both "sticky" and "balanced" assignor, and are tracking
this in it's own sub-task.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
