Siyao Meng created HDDS-17124:
---------------------------------
Summary: Leadership transfer is slow because
RatisHelper#transferRatisLeadership creates its RaftClient without a leader hint
Key: HDDS-17124
URL: https://issues.apache.org/jira/browse/HDDS-17124
Project: Apache Ozone
Issue Type: Sub-task
Components: OM HA, SCM HA
Reporter: Siyao Meng
{{RatisHelper#transferRatisLeadership}} (used by
{{OzoneManager#transferLeadership}} and
{{SCMClientProtocolServer#transferLeadership}}) builds a RaftClient with
{{leaderId = null}}. Two requests then usually go to a non-leader first, and
each miss costs a full Ratis client retry sleep:
# {{setConfiguration}} (priority change) is sent to the first peer of the
group. That peer is a follower with probability (N-1)/N, so 4/5 of the time in
a 5-node ring.
# {{resetPriorities}} runs after the transfer, when the client still has the
*old* leader cached, so it always misses once.
Measured in 5-node MiniOzoneCluster tests:
* OM: "Start transferring leadership" 17.520 -> setConfiguration first sent to
omNode-5 (follower) 17.570 -> reached the leader omNode-1 only at 24.088 (6.5
s); resetPriorities took another ~6 s.
* SCM: setConfiguration hit the leader first (fast), resetPriorities took 4.1 s.
So {{ozone admin om transfer}} / {{ozone admin scm transfer}} take ~4-13 s
instead of well under a second. With a short client RPC timeout this also
triggers the retry problem in the "OM transferLeadership is not idempotent"
sub-task.
*Proposal:* the caller always runs on the current leader, so pass the local
peer id as the client's leader, and send {{resetPriorities}} to the new leader
(the transfer target).
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
