Siyao Meng created HDDS-16832:
---------------------------------
Summary: OM that started as a single node fails every checkpoint
install from the first OM bootstrapped onto it with a NullPointerException
Key: HDDS-16832
URL: https://issues.apache.org/jira/browse/HDDS-16832
Project: Apache Ozone
Issue Type: Bug
Reporter: Siyao Meng
Attachments:
CR-6-add-first-bootstrapped-om-to-snapshot-provider.patch,
TestBugCR6OldOMCannotInstallCheckpointFromFirstNewOM.java
h3. Mechanism
An OM that starts with no peers has no {{OmRatisSnapshotProvider}}.
{{OzoneManager.addOMNodeToPeers}} creates it when the first OM is bootstrapped,
from {{peerNodesMap}} as it is before the new node is put in, and only the
{{else}} branch (the provider already exists) calls {{addNewPeerNode}}. The
provider keeps its own copy of the map, so the first bootstrapped OM is never
in it. OMs bootstrapped later are added.
{{OmRatisSnapshotProvider.downloadSnapshot}} uses the result of
{{peerNodesMap.get(leaderNodeID)}} without a null check. When the first
bootstrapped OM is leader and notifies the original OM to install a snapshot,
the lookup returns null and a {{NullPointerException}} leaves
{{OzoneManager.installSnapshotFromLeader}}, which catches only {{IOException}}.
Ratis logs "Failed to notify StateMachine to InstallSnapshot" and the leader
sends the notification again. Nothing adds the missing peer later, because
{{OzoneManager.updatePeerList}} only adds nodes that the Ratis server's peer
list does not have yet.
The {{addOMNodeToPeers}} block is the same in the release tags from 1.2.0 to
2.2.1 (read from the tagged source, not run there). It came in with HDDS-4330.
h3. Trigger
# One OM runs with a service ID as a single node ring.
# Two more OMs are bootstrapped. The existing OM is not restarted, which the OM
HA documentation says is not needed.
# The first bootstrapped OM becomes leader.
# The original OM falls behind the leader's purged log while its process stays
up (a partition or a long pause), so the leader notifies it to install a
snapshot.
h3. Impact
The original OM cannot catch up while the first bootstrapped OM leads. It stays
a member of the ring but no longer counts towards the commit majority, so the
ring cannot tolerate the loss of the other follower. Nothing reports this
except the ERROR lines in the original OM's log. There is no data loss.
Restarting the original OM is a workaround, because the provider is then built
from the configuration.
Run on a mini cluster with a short Raft log (purge gap and snapshot threshold
of 50): the original OM stayed at applied index 8 while the leader was at 102,
with 100 to 200 failed install attempts per second (2138 in 20 seconds in one
run, 3994 in 21 seconds in another), each one an ERROR line on the original OM.
A write with all three OMs up succeeded. With the second bootstrapped OM
stopped, a write did not complete within 20 seconds. After the original OM was
stopped and started again as a new {{OzoneManager}} from its configuration and
storage directory, it installed a checkpoint from the same leader and a write
succeeded.
Not exercised: default log settings (snapshot every 400,000 transactions, so
the OM has to miss far more), a real network partition, a secure cluster, and
the conversion of an OM that had no service ID.
h3. Reproduction
PASS in 2 of 2 runs on unmodified source.
[^TestBugCR6OldOMCannotInstallCheckpointFromFirstNewOM.java]
({{hadoop-ozone/integration-test}}) starts a mini cluster with one OM,
bootstraps two OMs, transfers leadership to the first and writes through the
client. The original OM is made to miss the transactions without a restart by
rejecting the {{appendEntries}} and {{installSnapshot}} calls it receives,
through the Ratis {{CodeInjectionForTesting}} hooks at the start of these
handlers. This stands for a partition and changes no state. The hooks are then
removed and the test observes the failed attempts, the two writes and the
restart. A passing test means the defect is present.
h3. Suggested fix
The attached patch removes the cause. A null check in
{{OmRatisSnapshotProvider.downloadSnapshot}} that throws an {{IOException}}
naming the unknown leader would in addition send any future missing peer
through the handled error path. It is not in the patch, since it makes no
install succeed.
h3. Patch
[^CR-6-add-first-bootstrapped-om-to-snapshot-provider.patch], against
7fcf31294859d5d31163e016b5a9a63f21c6edd2. It also applies to master at
1cc6423590f (not built there).
{{OzoneManager.addOMNodeToPeers}} now calls {{addNewPeerNode}} for the new OM
whether the provider already existed or was just created. For an OM that starts
with peers nothing changes.
Covered by an addition to the existing
{{TestAddRemoveOzoneManager.testBootstrap}}: after the two OMs are
bootstrapped, leadership moves to the first new OM, the old OM downloads a
checkpoint from it through its snapshot provider, and leadership moves back.
Without the change it fails with the {{NullPointerException}} above. With it
{{TestAddRemoveOzoneManager}}, {{TestOMInstallSnapshotDuringBootstrapping}} and
{{TestOzoneManagerSnapshotProvider}} pass (10 tests), as do
{{TestOzoneManagerStateMachine}}, {{TestOmRatisSnapshotProvider}} and
{{TestOzoneManagerRatisServer}} (74 tests), and checkstyle is clean. With the
patch the reproduction class fails at the assertion that expects the original
OM to stay behind, with 0 failed install attempts.
Found by code review of the OM HA membership change paths (bootstrap,
decommission and leader transfer), as part of the TLA+ verification effort
under HDDS-15926, on commit 7fcf31294859d5d31163e016b5a9a63f21c6edd2. Checked
against HDDS issues and apache/ozone pull requests for duplicates before
filing. The attached patch is a proposal for review. Generated with Specula
(Claude Opus 5.5).
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
