Siyao Meng created HDDS-16838:
---------------------------------
Summary: OM bootstrap check passes when the new OM's own config
lacks a ring member; the OM is added as a voter and then stops with an NPE
Key: HDDS-16838
URL: https://issues.apache.org/jira/browse/HDDS-16838
Project: Apache Ozone
Issue Type: Bug
Reporter: Siyao Meng
Attachments:
MC-6-check-bootstrapping-om-config-has-all-ring-members.patch,
TestBugMC6BootstrapOmittedMember.java
h3. Mechanism
{{OzoneManager.checkConfigBeforeBootstrap}} asks every OM in the new OM's own
config whether it knows the new OM. It never checks the other direction, that
the new OM's config has every member of the ring, although each existing OM
returns its peer list in the same reply
({{OMConfiguration.getCurrentPeerList}}).
A bootstrapping OM starts with an empty Ratis peer list and fills it from the
Raft configurations it is notified of. For a member without
{{ozone.om.address.<serviceId>.<nodeId>}} in the local config,
{{OzoneManager.addOMNodeToPeers}} logs "There is no OM configuration for node
ID ... in ozone-site.xml." and skips it. When the configuration that contains
the new OM itself is applied, the branch of {{OzoneManager.updatePeerList}} for
the local node calls
{{omRatisServer.addRaftPeer(peerNodesMap.get(peerNodeId))}} with null for that
member. The {{NullPointerException}} is thrown on the Ratis
{{StateMachineUpdater}} thread, Ratis closes the state machine and
{{OzoneManagerStateMachine.close}} terminates the OM with exit status 0. At
that point the leader has already committed the new OM as a voter and the new
OM has logged "Successfully bootstrapped OM".
Both the check and the unguarded {{addRaftPeer}} call are in the release tags
from 1.3.0 (HDDS-5534) to 2.2.1, and not in 1.2.x.
h3. Trigger
# Ring of omNode-1 (leader) and omNode-2. The new node is added to the config
of both.
# The {{ozone-site.xml}} of the new node has no entry for omNode-2 (not in
{{ozone.om.nodes.<serviceId>}} and no
{{ozone.om.address.<serviceId>.omNode-2}}).
# {{ozone om --bootstrap}} on the new node. The check passes because only
omNode-1 is asked. The leader adds the node, then the node stops with the NPE.
No fault and no {{--force}} is needed, but the config of the new node has to
differ from the one on the existing OMs, which the documented procedure (the
same updated {{ozone-site.xml}} on all OMs) does not produce. If only the nodes
list omits the member and its address key is present, the bootstrap works (run).
h3. Impact
* The new OM is a voter in the committed Raft configuration while its process
is down, after it logged a successful bootstrap and exited with status 0. A
ring of two becomes three voters with two running, so it keeps a leader but
tolerates no further failure (run, a write still succeeds).
* A second new OM started the same way, with a config that also lacks the first
new OM (otherwise the check fails because that OM is down), gives four voters
with two running. The leader steps down with {{LOST_MAJORITY_HEARTBEATS}}, no
leader is elected and a client write fails (run). On a ring of three the
numbers differ (not run): one such join leaves four voters with three running,
so the ring that tolerated one failure tolerates none, and it takes three such
joins (six voters, three running) to lose the leader.
* Recovery: starting a stopped OM normally with the complete config brings it
back. In the two join case a leader was ready within a few seconds of that
start on a mini cluster (0.5 to 4 s over five runs). A restart with the
unchanged config was not run.
h3. Reproduction
PASS, no fault injection and no timing hooks, unmodified source.
[^TestBugMC6BootstrapOmittedMember.java] ({{hadoop-ozone/integration-test}})
uses the real OMs of a mini cluster and the plain {{BOOTSTRAP}} startup option,
about 60 seconds. The new OM is created the way
{{MiniOzoneHAClusterImpl.bootstrapOzoneManager}} does it, except that its own
config differs from the config of the running OMs. The first test shows one
join (three members in the Raft configuration, the two log lines, the NPE, exit
status 0, and the recovery by a normal start). The second shows two joins
leaving the ring without a leader, a failed write, and the recovery. A passing
test means the defect is present. The first test passed in every run. The
second passed in five of six runs: on a busy host Ratis once refused the second
join ({{RATIS_BOOTSTRAP_ERROR}}, "Fail to set configuration ... due to
NOPROGRESS") because the new OM did not answer the leader within the short
timeouts of the mini cluster, and the test ended there with an error.
{noformat}
INFO om.OzoneManager - Successfully bootstrapped OM omNode-bootstrap-1 and
joined the Ratis group
ERROR impl.StateMachineUpdater -
[email protected] caught a Throwable.
java.lang.NullPointerException: Cannot invoke
"org.apache.hadoop.ozone.om.helpers.OMNodeDetails.getNodeId()" because
"omNodeDetails" is null
at
org.apache.hadoop.ozone.om.ratis.OzoneManagerRatisServer.addRaftPeer(OzoneManagerRatisServer.java:461)
at
org.apache.hadoop.ozone.om.OzoneManager.updatePeerList(OzoneManager.java:2338)
at
org.apache.hadoop.ozone.om.ratis.OzoneManagerStateMachine.notifyConfigurationChanged(OzoneManagerStateMachine.java:340)
INFO om.OzoneManager - Terminating with exit status 0: OM state machine is
shutdown by Ratis server
{noformat}
h3. Patch
[^MC-6-check-bootstrapping-om-config-has-all-ring-members.patch], against
7fcf31294859d5d31163e016b5a9a63f21c6edd2. It also applies to master at
1cc6423590f (not built there).
{{checkConfigBeforeBootstrap}} now also goes through the peer list each
existing OM reports ({{OMConfiguration.getCurrentPeerList}}, the in memory peer
list of that OM, not the Raft configuration) and exits with "OM(s) [...] are in
the peer list of the existing OMs but have no address in the configuration of
the bootstrapping OM. Update its ozone-site.xml before proceeding." before the
leader is asked. It uses only the address lookup that {{addOMNodeToPeers}}
starts with ({{OMNodeDetails.getOMNodeAddressFromConf}}), so a new OM whose
config has an address for every listed peer is not refused. An address that is
present but cannot be turned into a socket address is not covered and still
reaches the NPE (from the code, not run). No request or protocol change.
One bootstrap that works today is refused with the patch (from the code, not
run): an existing OM that was started with a node in its {{ozone.om.nodes}}
list that is not in the ring keeps that node in its in memory peer list until
it applies the next Raft configuration change, and a new OM whose config has no
address for that node is refused in that window. It needs the config files to
disagree.
Covered by {{testBootstrapWithExistingOMMissingInNewOMConfig}} in the existing
{{TestAddRemoveOzoneManager}}, which also asserts that no existing OM has the
new OM in its Raft configuration afterwards. Without the change it fails
because {{start()}} does not throw. With it the nine tests of
{{TestAddRemoveOzoneManager}} pass, checkstyle is clean, and both tests of the
reproduction fail at their first bootstrap with the new message.
Not changed: {{--force}} skips the whole check, so the NPE can still be reached
with it. A null check in {{updatePeerList}} alone would keep the OM running
without the member in its peer lists, so it is not part of this patch.
Found by TLA+ model checking and code review of the OM HA membership change
paths (bootstrap, decommission and leader transfer) under HDDS-15926, on commit
7fcf31294859d5d31163e016b5a9a63f21c6edd2. Checked against HDDS issues and
apache/ozone pull requests for duplicates before filing. The attached patch is
a proposal for review. Generated with Specula (Claude Opus 5.5).
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
