[
https://issues.apache.org/jira/browse/HDDS-16832?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Siyao Meng updated HDDS-16832:
------------------------------
Attachment: TestBugCR6OldOMCannotInstallCheckpointFromFirstNewOM.java
> OM that started as a single node fails every checkpoint install from the
> first OM bootstrapped onto it with a NullPointerException
> ----------------------------------------------------------------------------------------------------------------------------------
>
> Key: HDDS-16832
> URL: https://issues.apache.org/jira/browse/HDDS-16832
> Project: Apache Ozone
> Issue Type: Bug
> Reporter: Siyao Meng
> Priority: Major
> Attachments:
> CR-6-add-first-bootstrapped-om-to-snapshot-provider.patch,
> TestBugCR6OldOMCannotInstallCheckpointFromFirstNewOM.java
>
>
> h3. Mechanism
> An OM that starts with no peers has no {{OmRatisSnapshotProvider}}.
> {{OzoneManager.addOMNodeToPeers}} creates it when the first OM is
> bootstrapped, from {{peerNodesMap}} as it is before the new node is put in,
> and only the {{else}} branch (the provider already exists) calls
> {{addNewPeerNode}}. The provider keeps its own copy of the map, so the first
> bootstrapped OM is never in it. OMs bootstrapped later are added.
> {{OmRatisSnapshotProvider.downloadSnapshot}} uses the result of
> {{peerNodesMap.get(leaderNodeID)}} without a null check. When the first
> bootstrapped OM is leader and notifies the original OM to install a snapshot,
> the lookup returns null and a {{NullPointerException}} leaves
> {{OzoneManager.installSnapshotFromLeader}}, which catches only
> {{IOException}}. Ratis logs "Failed to notify StateMachine to
> InstallSnapshot" and the leader sends the notification again. Nothing adds
> the missing peer later, because {{OzoneManager.updatePeerList}} only adds
> nodes that the Ratis server's peer list does not have yet.
> The {{addOMNodeToPeers}} block is the same in the release tags from 1.2.0 to
> 2.2.1 (read from the tagged source, not run there). It came in with HDDS-4330.
> h3. Trigger
> # One OM runs with a service ID as a single node ring.
> # Two more OMs are bootstrapped. The existing OM is not restarted, which the
> OM HA documentation says is not needed.
> # The first bootstrapped OM becomes leader.
> # The original OM falls behind the leader's purged log while its process
> stays up (a partition or a long pause), so the leader notifies it to install
> a snapshot.
> h3. Impact
> The original OM cannot catch up while the first bootstrapped OM leads. It
> stays a member of the ring but no longer counts towards the commit majority,
> so the ring cannot tolerate the loss of the other follower. Nothing reports
> this except the ERROR lines in the original OM's log. There is no data loss.
> Restarting the original OM is a workaround, because the provider is then
> built from the configuration.
> Run on a mini cluster with a short Raft log (purge gap and snapshot threshold
> of 50): the original OM stayed at applied index 8 while the leader was at
> 102, with 100 to 200 failed install attempts per second (2138 in 20 seconds
> in one run, 3994 in 21 seconds in another), each one an ERROR line on the
> original OM. A write with all three OMs up succeeded. With the second
> bootstrapped OM stopped, a write did not complete within 20 seconds. After
> the original OM was stopped and started again as a new {{OzoneManager}} from
> its configuration and storage directory, it installed a checkpoint from the
> same leader and a write succeeded.
> Not exercised: default log settings (snapshot every 400,000 transactions, so
> the OM has to miss far more), a real network partition, a secure cluster, and
> the conversion of an OM that had no service ID.
> h3. Reproduction
> PASS in 2 of 2 runs on unmodified source.
> [^TestBugCR6OldOMCannotInstallCheckpointFromFirstNewOM.java]
> ({{hadoop-ozone/integration-test}}) starts a mini cluster with one OM,
> bootstraps two OMs, transfers leadership to the first and writes through the
> client. The original OM is made to miss the transactions without a restart by
> rejecting the {{appendEntries}} and {{installSnapshot}} calls it receives,
> through the Ratis {{CodeInjectionForTesting}} hooks at the start of these
> handlers. This stands for a partition and changes no state. The hooks are
> then removed and the test observes the failed attempts, the two writes and
> the restart. A passing test means the defect is present.
> h3. Suggested fix
> The attached patch removes the cause. A null check in
> {{OmRatisSnapshotProvider.downloadSnapshot}} that throws an {{IOException}}
> naming the unknown leader would in addition send any future missing peer
> through the handled error path. It is not in the patch, since it makes no
> install succeed.
> h3. Patch
> [^CR-6-add-first-bootstrapped-om-to-snapshot-provider.patch], against
> 7fcf31294859d5d31163e016b5a9a63f21c6edd2. It also applies to master at
> 1cc6423590f (not built there).
> {{OzoneManager.addOMNodeToPeers}} now calls {{addNewPeerNode}} for the new OM
> whether the provider already existed or was just created. For an OM that
> starts with peers nothing changes.
> Covered by an addition to the existing
> {{TestAddRemoveOzoneManager.testBootstrap}}: after the two OMs are
> bootstrapped, leadership moves to the first new OM, the old OM downloads a
> checkpoint from it through its snapshot provider, and leadership moves back.
> Without the change it fails with the {{NullPointerException}} above. With it
> {{TestAddRemoveOzoneManager}}, {{TestOMInstallSnapshotDuringBootstrapping}}
> and {{TestOzoneManagerSnapshotProvider}} pass (10 tests), as do
> {{TestOzoneManagerStateMachine}}, {{TestOmRatisSnapshotProvider}} and
> {{TestOzoneManagerRatisServer}} (74 tests), and checkstyle is clean. With the
> patch the reproduction class fails at the assertion that expects the original
> OM to stay behind, with 0 failed install attempts.
> Found by code review of the OM HA membership change paths (bootstrap,
> decommission and leader transfer), as part of the TLA+ verification effort
> under HDDS-15926, on commit 7fcf31294859d5d31163e016b5a9a63f21c6edd2. Checked
> against HDDS issues and apache/ozone pull requests for duplicates before
> filing. The attached patch is a proposal for review. Generated with Specula
> (Claude Opus 5.5).
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
