[
https://issues.apache.org/jira/browse/HDDS-16833?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Siyao Meng updated HDDS-16833:
------------------------------
Attachment: TestBugMC3TransferRevertsMembershipChange.java
> Leader transfer reverts a concurrent OM or SCM membership change because it
> rewrites the Raft configuration from a group read earlier
> -------------------------------------------------------------------------------------------------------------------------------------
>
> Key: HDDS-16833
> URL: https://issues.apache.org/jira/browse/HDDS-16833
> Project: Apache Ozone
> Issue Type: Bug
> Reporter: Siyao Meng
> Priority: Major
> Attachments:
> MC-3-compare-and-set-priorities-in-leader-transfer.patch,
> TestBugMC3TransferRevertsMembershipChange.java
>
>
> h3. Mechanism
> {{RatisHelper.transferRatisLeadership}} takes the Raft group read by its
> caller, compares it once with the group reported by the transfer target, and
> then calls Ratis {{setConfiguration}} twice with peer lists built from that
> group: once to raise the priority of the target, and once in
> {{resetPriorities}} after the transfer. Both calls use the default mode
> ({{SET_UNCONDITIONALLY}}), which replaces the whole configuration. A
> membership change that commits after the group was read and before one of the
> two requests reaches the leader is overwritten: a newly added node is
> dropped, a decommissioned node is added back.
> The window before the reset is open on every successful transfer. The Ratis
> client still has the old leader as its leader, so it sends the reset there
> first and gets {{NotLeaderException}} in the reply. For an exception that
> arrives in a reply, {{BlockingImpl.sendRequestWithRetry}} sleeps for the
> interval of the retry policy before it retries on the new leader. With the
> default {{hdds.ratis.client.multilinear.random.retry.policy}} ("5s, 6") that
> is 2.5 to 7.5 seconds, during which the new leader accepts membership
> changes. The window before the priority change opens the same way when the
> first peer the client contacts is not the leader, and it also stays open
> while another reconfiguration is in progress, because
> {{ReconfigurationInProgressException}} is retried with the same list. Ratis
> rejects configuration changes while the leader is stepping down, so the
> transfer itself is not part of either window.
> The helper has two callers, {{OzoneManager.transferLeadership}} and
> {{SCMClientProtocolServer.transferLeadership}}. No datanode or container code
> uses it.
> h3. Trigger
> Two admin operations overlap on a ring of three OMs. No fault is needed.
> # {{ozone admin om transfer -n om2}} is running. The leadership has moved to
> om2 and the priority reset is waiting for its retry.
> # An OM bootstrap of a new node om4, or {{ozone admin om decommission}} of
> om3, commits on om2 and reports success.
> # The retried reset installs the peers that were read before step 2.
> For the decommission direction the decommissioned OM must still be running
> when the stale list arrives, otherwise Ratis cannot add it back and the
> request fails with the membership unchanged. A removed OM stops itself about
> one election timeout after its removal ({{ozone.om.ratis.minimum.timeout}},
> default 5s).
> h3. Impact
> * Bootstrap: the new OM logs "Successfully bootstrapped OM", is then removed
> from the Raft configuration and shuts itself down. The ring is back at its
> previous size although the bootstrap reported success.
> * Decommission: the command reports success, and the decommissioned OM is a
> voter again in the Raft configuration while {{ozone.om.decommissioned.nodes}}
> lists it as decommissioned.
> * Nothing repairs either outcome. No data is lost, and repeating the
> bootstrap or the decommission works.
> * SCM: {{ozone admin scm transfer}} uses the same helper, so an SCM added or
> decommissioned in the same windows would be reverted the same way. This is
> argued from the shared code, not run.
> h3. Reproduction
> PASS, deterministic, unmodified source.
> [^TestBugMC3TransferRevertsMembershipChange.java]
> ({{hadoop-ozone/integration-test}}) runs both admin operations through the
> real OM entry points on a three OM mini cluster. It replaces only the retry
> sleep of the transfer client: the old leader gets a retry policy
> ({{hdds.ratis.client.retry.policy}}) that runs the second admin operation and
> then retries at once, which fixes the order of events without a sleep or a
> failpoint. In the first test the bootstrapped OM is no longer in the Raft
> configuration after the transfer returns, and it stops. In the second test
> the decommission returns success with two voters left, and after the transfer
> returns the decommissioned OM is a voter again. A passing test means the
> defect is present.
> Without the test policy, starting a bootstrap as soon as the leadership had
> moved reverted the bootstrap in 2 of 2 runs with default settings (run
> separately, not part of the attached test).
> h3. Suggested fix
> The attached patch keeps the current sequence and makes both requests
> conditional. A larger alternative is to stop changing priorities: in Ratis
> 3.3.1 {{transferLeadership}} only requires that no peer has a higher priority
> than the target, which holds when all priorities are equal (HDDS-9627 already
> mentions this as a follow up). That would remove both {{setConfiguration}}
> calls, but it changes the transfer for every cluster and needs a fallback for
> a ring that has unequal priorities.
> h3. Patch
> [^MC-3-compare-and-set-priorities-in-leader-transfer.patch], against
> 7fcf31294859d5d31163e016b5a9a63f21c6edd2. It also applies to master at
> 1cc6423590f.
> Both requests now use the Ratis {{COMPARE_AND_SET}} mode with the peers of
> the request as the expected current members. Ratis compares the peer ids of
> voters and listeners with its current configuration under the same lock that
> starts the reconfiguration, so a list built from an outdated group fails with
> {{SetConfigurationException}} instead of being applied. If the membership
> changes before the priority change, the transfer command fails with that
> error and nothing is changed. If it changes before the reset, the transfer
> has already happened, the reset is skipped with the existing "Failed to reset
> priorities" error log, and the membership change is kept. A transfer that
> does not overlap with a membership change behaves as before.
> One thing is left open on purpose. When the reset is skipped, the priorities
> are whatever the membership change wrote. For OM that is the default priority
> for every peer, because OM bootstrap and decommission write the default
> priority for every member. For SCM, {{addSCM}} and {{removeSCM}} copy the
> current peers, so the target keeps its raised priority until the next
> transfer. Reading the group again and repeating the reset would close that,
> at the cost of a larger change.
> Covered by {{testBootstrapDuringLeaderTransfer}} in the existing
> {{TestAddRemoveOzoneManager}}, on a ring of three voters and one listener,
> which also checks that every peer has the default priority afterwards.
> Without the change it fails because the bootstrapped OM is missing from the
> Raft configuration of the leader. With it the test passes, as do
> {{TestAddRemoveOzoneManager}} (9 tests), {{TestTransferLeadershipShell}} (5
> tests, OM and SCM transfers including the priority reset check),
> {{TestOMHALeaderSpecificACLEnforcement}},
> {{TestSCMFollowerCatchupWithContainerReport}},
> {{TestDatanodeSCMNodesReconfiguration}} and {{TestRatisHelper}}, and
> checkstyle is clean. With the patch both tests of the reproduction fail,
> because the membership change is kept.
> Found by TLA+ model checking and code review of the OM HA membership change
> paths (bootstrap, decommission and leader transfer) under HDDS-15926, on
> commit 7fcf31294859d5d31163e016b5a9a63f21c6edd2. Checked against HDDS issues
> and apache/ozone pull requests for duplicates before filing. The attached
> patch is a proposal for review. Generated with Specula (Claude Opus 5.5).
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
