Siyao Meng created HDDS-17125:
---------------------------------
Summary: OM transferLeadership is not idempotent under client
failover retry
Key: HDDS-17125
URL: https://issues.apache.org/jira/browse/HDDS-17125
Project: Apache Ozone
Issue Type: Sub-task
Components: OM HA
Reporter: Siyao Meng
If the {{transferLeadership}} admin RPC takes longer than
{{ozone.om.client.rpc.timeout}} (see the RatisHelper leader hint sub-task for
why it can take 4-13 s), the OM client fails over and sends the same request to
the next OM. After a successful transfer that is the *new* leader, which then
runs a second set-priority / transfer / reset-priority cycle while the first
one is still resetting priorities.
Observed in a 5-node test (5 s client RPC timeout):
* 17.52 omNode-1 starts transfer to omNode-2; 22.52 client
SocketTimeoutException and failover.
* 24.21 transfer to omNode-2 succeeds; 24.55 the retried request arrives at
omNode-2: "Start transferring leadership to omNode-2" (to itself).
* 29.83 / 30.10 both handlers reset priorities; 31.60 omNode-2 steps down and
omNode-4 is elected. That is an extra leader change nobody asked for.
* Client fails with "Could not determine or connect to OM Leader" after 10
failovers.
*Proposal:* treat "target is already the leader" as success without touching
priorities, and/or do not retry {{transferLeadership}} on a different OM after
a timeout.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
