[
https://issues.apache.org/jira/browse/HDDS-17124?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Siyao Meng updated HDDS-17124:
------------------------------
Description:
{{RatisHelper#transferRatisLeadership}} (used by
{{OzoneManager#transferLeadership}} and
{{SCMClientProtocolServer#transferLeadership}}) builds a RaftClient with
{{leaderId = null}}. Two requests then usually go to a non-leader first, and
each miss costs a full Ratis client retry sleep:
# {{setConfiguration}} (priority change) is sent to the first peer of the
group. That peer is a follower with probability (N-1)/N, so 4/5 of the time in
a 5-node ring.
# {{resetPriorities}} runs after the transfer, when the client still has the
*old* leader cached, so it always misses once.
Measured in 5-node MiniOzoneCluster tests:
* OM: "Start transferring leadership" 17.520 -> setConfiguration first sent to
omNode-5 (follower) 17.570 -> reached the leader omNode-1 only at 24.088 (6.5
s); resetPriorities took another ~6 s.
* SCM: setConfiguration hit the leader first (fast), resetPriorities took 4.1 s.
So {{ozone admin om transfer}} / {{ozone admin scm transfer}} take ~4-13 s
instead of well under a second. With a short client RPC timeout this also
triggers the retry problem in HDDS-17125.
*Proposal:* the caller always runs on the current leader, so pass the local
peer id as the client's leader, and send {{resetPriorities}} to the new leader
(the transfer target).
was:
{{RatisHelper#transferRatisLeadership}} (used by
{{OzoneManager#transferLeadership}} and
{{SCMClientProtocolServer#transferLeadership}}) builds a RaftClient with
{{leaderId = null}}. Two requests then usually go to a non-leader first, and
each miss costs a full Ratis client retry sleep:
# {{setConfiguration}} (priority change) is sent to the first peer of the
group. That peer is a follower with probability (N-1)/N, so 4/5 of the time in
a 5-node ring.
# {{resetPriorities}} runs after the transfer, when the client still has the
*old* leader cached, so it always misses once.
Measured in 5-node MiniOzoneCluster tests:
* OM: "Start transferring leadership" 17.520 -> setConfiguration first sent to
omNode-5 (follower) 17.570 -> reached the leader omNode-1 only at 24.088 (6.5
s); resetPriorities took another ~6 s.
* SCM: setConfiguration hit the leader first (fast), resetPriorities took 4.1 s.
So {{ozone admin om transfer}} / {{ozone admin scm transfer}} take ~4-13 s
instead of well under a second. With a short client RPC timeout this also
triggers the retry problem in the "OM transferLeadership is not idempotent"
sub-task.
*Proposal:* the caller always runs on the current leader, so pass the local
peer id as the client's leader, and send {{resetPriorities}} to the new leader
(the transfer target).
> Leadership transfer is slow because RatisHelper#transferRatisLeadership
> creates its RaftClient without a leader hint
> --------------------------------------------------------------------------------------------------------------------
>
> Key: HDDS-17124
> URL: https://issues.apache.org/jira/browse/HDDS-17124
> Project: Apache Ozone
> Issue Type: Sub-task
> Components: OM HA, SCM HA
> Reporter: Siyao Meng
> Priority: Major
>
> {{RatisHelper#transferRatisLeadership}} (used by
> {{OzoneManager#transferLeadership}} and
> {{SCMClientProtocolServer#transferLeadership}}) builds a RaftClient with
> {{leaderId = null}}. Two requests then usually go to a non-leader first, and
> each miss costs a full Ratis client retry sleep:
> # {{setConfiguration}} (priority change) is sent to the first peer of the
> group. That peer is a follower with probability (N-1)/N, so 4/5 of the time
> in a 5-node ring.
> # {{resetPriorities}} runs after the transfer, when the client still has the
> *old* leader cached, so it always misses once.
> Measured in 5-node MiniOzoneCluster tests:
> * OM: "Start transferring leadership" 17.520 -> setConfiguration first sent
> to omNode-5 (follower) 17.570 -> reached the leader omNode-1 only at 24.088
> (6.5 s); resetPriorities took another ~6 s.
> * SCM: setConfiguration hit the leader first (fast), resetPriorities took 4.1
> s.
> So {{ozone admin om transfer}} / {{ozone admin scm transfer}} take ~4-13 s
> instead of well under a second. With a short client RPC timeout this also
> triggers the retry problem in HDDS-17125.
> *Proposal:* the caller always runs on the current leader, so pass the local
> peer id as the client's leader, and send {{resetPriorities}} to the new
> leader (the transfer target).
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
