[
https://issues.apache.org/jira/browse/HDDS-16839?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Siyao Meng updated HDDS-16839:
------------------------------
Attachment: CR-3-wait-for-membership-after-failed-bootstrap-request.patch
> A new OM treats any failed bootstrap reply as final and exits, although the
> leader may still be adding it or may already have logged the change
> -----------------------------------------------------------------------------------------------------------------------------------------------
>
> Key: HDDS-16839
> URL: https://issues.apache.org/jira/browse/HDDS-16839
> Project: Apache Ozone
> Issue Type: Bug
> Reporter: Siyao Meng
> Priority: Minor
> Attachments:
> CR-3-wait-for-membership-after-failed-bootstrap-request.patch,
> TestBugCR3BootstrapFailsButChangeCommits.java
>
>
> h3. Mechanism
> {{OzoneManager.bootstrap}} sends one {{BootstrapOMRequest}} and ends the
> process on any exception. The outcome of the request and the outcome of the
> membership change are not the same thing in two cases.
> # The leader answers {{success=false}} whenever
> {{OzoneManagerRatisServer.addOMToRatisRing}} throws, which includes losing
> leadership after the configuration entry that adds the new OM is already in
> the Raft log. The next leader can still commit that entry.
> # {{OMInterServiceProtocolClientSideImpl}} retries the request through the
> failover proxy when the RPC itself fails (for example on
> {{ozone.om.client.rpc.timeout}} or a lost connection). The first request
> keeps running on the leader, so the retry is answered with
> {{RATIS_BOOTSTRAP_ERROR}} "Reconfiguration is already in progress", which is
> not retried.
> In both cases the new OM logs "Failed to Bootstrap OM." and exits. It never
> looks at the Raft configuration it receives, although it is told about the
> committed change through
> {{OzoneManagerStateMachine.notifyConfigurationChanged}} like every other
> member.
> h3. Trigger
> Case 1 (run with timing help from the Ratis test hook, see Reproduction):
> # Three OMs. The new OM is started with {{ozone om --bootstrap}} and catches
> up.
> # The leader logs the configuration entry. One follower and the new OM are
> slow, so the entry cannot be committed, the leader loses its majority and
> steps down.
> # The pending change is answered with a failure and the new OM exits.
> # The slow follower recovers, a new leader is elected and commits the entry.
> Without the hook the leader has to lose leadership or stop between appending
> the first configuration entry of the change, which it does only after the new
> OM has caught up, and committing the last one. That is two commit round
> trips, a few milliseconds per bootstrap in a healthy ring (from the code).
> The test hook holds this window open.
> Case 2 (run without any hook): the first bootstrap request of the new OM
> times out on the new OM while the leader is still adding it, with
> {{ozone.om.client.rpc.timeout}} lowered to 200 ms in the test. The retry
> fails with "Reconfiguration is already in progress" and the new OM gives up
> about a quarter of a second after its first request reached the leader. With
> the default timeout of 15 minutes this needs a leader that takes longer than
> that to bring the new OM up to date, or a connection failure during that time
> (argued, not run).
> h3. Impact
> * Case 1: the bootstrap command fails and the process of the new OM ends, but
> the new OM is a voter in the committed Raft configuration of every existing
> OM. A ring of three becomes four voters with three running, so it tolerates
> no further failure until somebody notices and starts the OM. Starting it
> normally (without {{--bootstrap}}) is the way out; I did not run that step.
> * Case 2: a bootstrap that the leader was in the middle of completing is
> reported as failed. What the leader does after the new OM is gone was not run.
> h3. Reproduction
> PASS on unmodified source. [^TestBugCR3BootstrapFailsButChangeCommits.java]
> ({{hadoop-ozone/integration-test}}) shows case 1 with the real OMs of a mini
> cluster and its normal bootstrap. It uses the Ratis test hook
> {{CodeInjectionForTesting}} to delay the handling of AppendEntries on one
> follower and, from the entry that adds it, on the new OM. Only timing is
> changed. The leader steps down with {{LOST_MAJORITY_HEARTBEATS}}, the
> bootstrap fails with {{RATIS_BOOTSTRAP_ERROR}} "Failed to add OM
> omNode-bootstrap-1 in Ratis ring.", and one to two seconds later, after a new
> leader is elected, all three existing OMs have a stable Raft configuration
> with four voters that include the new OM, whose process has ended. A passing
> test means the defect is present.
> Case 2 is the first regression test in the patch.
> h3. Suggested fix
> The attached patch changes only the new OM. It narrows the window and does
> not close it.
> * If the change is committed later than the wait, the new OM still exits. In
> case 2 it now exits at the end of the wait instead of at once, so a leader
> that finishes just then leaves the voter of case 1 (from the code, not run).
> * In case 2 at the default {{ozone.om.client.rpc.timeout}} the first request
> has already run for 15 minutes when the retry is refused, so the patch helps
> only if the leader finishes within the wait that follows. Raising
> {{ozone.om.client.rpc.timeout}} in the config of the new OM avoids the retry
> in the first place (from the code, not run).
> * The patch can also report success too early. The condition reads only the
> state of the new OM, and it depends on the leader handing its current
> configuration to a new peer before the log. Ratis 3.3.1 does that with the
> snapshot notification the OM uses, once for each peer the leader starts to
> bring up to date; it is a property of that implementation, not of the Ratis
> API. A process that is started again with an empty disk while the leader is
> still bringing the earlier process of the same OM up to date gets the log
> from the start without that configuration. If its node ID was a member before
> and the leader's log still has those entries, the wait can end for the old
> configuration while the leader has not logged the new change. Run on a mini
> cluster: the second process reported a completed bootstrap while the latest
> entry in the leader's log was the old removal. Forced in that run: one log
> entry per request ({{ozone.om.ratis.log.appender.queue.num-elements=1}}), the
> second process started 0.6 s after the first had given up, and the old
> removal held back with the Ratis test hook. Not run without those. The leader
> is still adding the OM at that point, so the report is early, and wrong only
> if that attempt fails afterwards.
> The alternative to prefer changes the leader side.
> {{OzoneManagerRatisServer.addOMToRatisRing}} adds the new OM to the peer list
> without looking whether it is already a member. If it answered a repeated add
> of a member with success (Ratis already answers a request that changes
> nothing with success, once no change is in progress), a bounded retry in
> {{OzoneManager.bootstrap}} would take the answer from the leader, and the new
> OM would have nothing to infer. From the code, not run and not implemented
> here. The attached patch is what the new OM can do alone.
> h3. Patch
> [^CR-3-wait-for-membership-after-failed-bootstrap-request.patch], against
> 7fcf31294859d5d31163e016b5a9a63f21c6edd2. It also applies to master at
> 1cc6423590f (not built there).
> After a failed request {{OzoneManager.bootstrap}} logs a warning with the
> cause and the bound, and waits for at most
> {{ozone.om.ratis.server.failure.timeout.duration}} (default 120s) until the
> latest Raft configuration the new OM knows has the OM itself as a voter or
> listener, has been applied locally, and is newer than the configuration the
> OM had before the request and not older than any configuration it has been
> notified of. If that happens the OM logs a second warning and continues its
> startup, otherwise it fails as before. A bootstrap that fails for good is
> therefore reported up to that much later, which is noted in the description
> of the key in {{ozone-default.xml}}. No request or protocol change.
> The last two parts of the condition are for a node ID that was a member
> before. A new OM with an empty disk can replay old configurations that have
> it. The replay starts after the last transaction of the checkpoint it gets
> from the leader, so this needs a ring without writes since that membership
> (from the code, not run with a write). The leader's current configuration,
> which Ratis 3.3.1 hands to the new OM before the log, is dropped again by
> Ratis when the log is replayed in more than one request.
> {{OzoneManagerStateMachine}} therefore keeps the highest configuration index
> it has been notified of.
> Run: with a condition on the Ratis peer list of the OM, such an OM reports a
> completed bootstrap for a change that is not committed. With a condition on
> the latest configuration being applied alone it does so when the old log
> reaches it one entry per request
> ({{ozone.om.ratis.log.appender.queue.num-elements=1}}). With the default of
> 1024 the leader sent the old entries in one request and the start failed
> correctly.
> Covered in the existing {{TestAddRemoveOzoneManager}}:
> * {{testBootstrapWhenRequestFailsWhileOMIsAdded}} (case 2, a voter and then a
> listener). Without the change it fails with "Failed to Bootstrap OM. Error
> Code: RATIS_BOOTSTRAP_ERROR, Error Message: Reconfiguration is already in
> progress".
> * {{testBootstrapFailsWhenOnlyOldConfigurationHasOM}}: an OM is bootstrapped,
> decommissioned and bootstrapped again with an empty disk, its request fails,
> and the log entries from its removal on are held back with the Ratis test
> hook. The bootstrap has to fail after the wait with the original exception,
> while the latest configuration of the new OM is the old one that has it.
> With the change the ten tests of {{TestAddRemoveOzoneManager}} pass and
> checkstyle is clean. With the patch the reproduction fails at its first
> assertion because the bootstrap returns normally: the new OM logs the second
> warning one to two seconds after the failed reply, once the new leader has
> committed the change.
> Found by code review of the OM HA membership change paths (bootstrap,
> decommission and leader transfer), as part of the TLA+ verification effort
> under HDDS-15926, on commit 7fcf31294859d5d31163e016b5a9a63f21c6edd2. Checked
> against HDDS issues and apache/ozone pull requests for duplicates before
> filing. The attached patch is a proposal for review. Generated with Specula
> (Claude Opus 5.5).
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
