Siyao Meng created HDDS-17126:
---------------------------------
Summary: OM client cannot fail over to a suggested leader that is
missing from its ozone.om.nodes configuration
Key: HDDS-17126
URL: https://issues.apache.org/jira/browse/HDDS-17126
Project: Apache Ozone
Issue Type: Sub-task
Components: Ozone Client, OM HA
Reporter: Siyao Meng
{{OMFailoverProxyProviderBase#getRetryAction}} follows the leader suggested in
{{OMNotLeaderException}} only if {{omProxies.contains(suggestedNodeId,
suggestedLeaderAddress)}}, even though the exception carries the leader's
address. After an OM ring grows (e.g. 3 -> 5 with {{ozone om --bootstrap}}),
any client still configured with the old OM list (Ozone clients, S3 Gateway,
Recon, HttpFS, admin tools) cannot reach the ring once a new OM becomes leader.
It cycles through the old OMs and fails, e.g. with ConnectException when the
old ones are down.
Reproduced in
{{TestAddRemoveOzoneManager#testBootstrapToFiveOMsAndDecommissionToThree}}:
after bootstrapping 2 OMs and stopping 2 original ones, the client built from
the 3-OM configuration failed with "Call From ... to localhost:15024 failed on
connection exception".
Datanodes can already pick up SCM node changes through reconfiguration
({{ozone.scm.nodes.<serviceId>}}), but OM clients have no equivalent.
*Options:* add the suggested leader to the proxy list dynamically, or document
clearly that every client configuration must be updated before a new OM can
become leader (and consider keeping new OMs from leading until then, e.g. via
priority).
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
