Siyao Meng created HDDS-16833:
---------------------------------
Summary: Leader transfer reverts a concurrent OM or SCM membership
change because it rewrites the Raft configuration from a group read earlier
Key: HDDS-16833
URL: https://issues.apache.org/jira/browse/HDDS-16833
Project: Apache Ozone
Issue Type: Bug
Reporter: Siyao Meng
Attachments: MC-3-compare-and-set-priorities-in-leader-transfer.patch,
TestBugMC3TransferRevertsMembershipChange.java
h3. Mechanism
{{RatisHelper.transferRatisLeadership}} takes the Raft group read by its
caller, compares it once with the group reported by the transfer target, and
then calls Ratis {{setConfiguration}} twice with peer lists built from that
group: once to raise the priority of the target, and once in
{{resetPriorities}} after the transfer. Both calls use the default mode
({{SET_UNCONDITIONALLY}}), which replaces the whole configuration. A membership
change that commits after the group was read and before one of the two requests
reaches the leader is overwritten: a newly added node is dropped, a
decommissioned node is added back.
The window before the reset is open on every successful transfer. The Ratis
client still has the old leader as its leader, so it sends the reset there
first and gets {{NotLeaderException}} in the reply. For an exception that
arrives in a reply, {{BlockingImpl.sendRequestWithRetry}} sleeps for the
interval of the retry policy before it retries on the new leader. With the
default {{hdds.ratis.client.multilinear.random.retry.policy}} ("5s, 6") that is
2.5 to 7.5 seconds, during which the new leader accepts membership changes. The
window before the priority change opens the same way when the first peer the
client contacts is not the leader, and it also stays open while another
reconfiguration is in progress, because {{ReconfigurationInProgressException}}
is retried with the same list. Ratis rejects configuration changes while the
leader is stepping down, so the transfer itself is not part of either window.
The helper has two callers, {{OzoneManager.transferLeadership}} and
{{SCMClientProtocolServer.transferLeadership}}. No datanode or container code
uses it.
h3. Trigger
Two admin operations overlap on a ring of three OMs. No fault is needed.
# {{ozone admin om transfer -n om2}} is running. The leadership has moved to
om2 and the priority reset is waiting for its retry.
# An OM bootstrap of a new node om4, or {{ozone admin om decommission}} of om3,
commits on om2 and reports success.
# The retried reset installs the peers that were read before step 2.
For the decommission direction the decommissioned OM must still be running when
the stale list arrives, otherwise Ratis cannot add it back and the request
fails with the membership unchanged. A removed OM stops itself about one
election timeout after its removal ({{ozone.om.ratis.minimum.timeout}}, default
5s).
h3. Impact
* Bootstrap: the new OM logs "Successfully bootstrapped OM", is then removed
from the Raft configuration and shuts itself down. The ring is back at its
previous size although the bootstrap reported success.
* Decommission: the command reports success, and the decommissioned OM is a
voter again in the Raft configuration while {{ozone.om.decommissioned.nodes}}
lists it as decommissioned.
* Nothing repairs either outcome. No data is lost, and repeating the bootstrap
or the decommission works.
* SCM: {{ozone admin scm transfer}} uses the same helper, so an SCM added or
decommissioned in the same windows would be reverted the same way. This is
argued from the shared code, not run.
h3. Reproduction
PASS, deterministic, unmodified source.
[^TestBugMC3TransferRevertsMembershipChange.java]
({{hadoop-ozone/integration-test}}) runs both admin operations through the real
OM entry points on a three OM mini cluster. It replaces only the retry sleep of
the transfer client: the old leader gets a retry policy
({{hdds.ratis.client.retry.policy}}) that runs the second admin operation and
then retries at once, which fixes the order of events without a sleep or a
failpoint. In the first test the bootstrapped OM is no longer in the Raft
configuration after the transfer returns, and it stops. In the second test the
decommission returns success with two voters left, and after the transfer
returns the decommissioned OM is a voter again. A passing test means the defect
is present.
Without the test policy, starting a bootstrap as soon as the leadership had
moved reverted the bootstrap in 2 of 2 runs with default settings (run
separately, not part of the attached test).
h3. Suggested fix
The attached patch keeps the current sequence and makes both requests
conditional. A larger alternative is to stop changing priorities: in Ratis
3.3.1 {{transferLeadership}} only requires that no peer has a higher priority
than the target, which holds when all priorities are equal (HDDS-9627 already
mentions this as a follow up). That would remove both {{setConfiguration}}
calls, but it changes the transfer for every cluster and needs a fallback for a
ring that has unequal priorities.
h3. Patch
[^MC-3-compare-and-set-priorities-in-leader-transfer.patch], against
7fcf31294859d5d31163e016b5a9a63f21c6edd2. It also applies to master at
1cc6423590f.
Both requests now use the Ratis {{COMPARE_AND_SET}} mode with the peers of the
request as the expected current members. Ratis compares the peer ids of voters
and listeners with its current configuration under the same lock that starts
the reconfiguration, so a list built from an outdated group fails with
{{SetConfigurationException}} instead of being applied. If the membership
changes before the priority change, the transfer command fails with that error
and nothing is changed. If it changes before the reset, the transfer has
already happened, the reset is skipped with the existing "Failed to reset
priorities" error log, and the membership change is kept. A transfer that does
not overlap with a membership change behaves as before.
One thing is left open on purpose. When the reset is skipped, the priorities
are whatever the membership change wrote. For OM that is the default priority
for every peer, because OM bootstrap and decommission write the default
priority for every member. For SCM, {{addSCM}} and {{removeSCM}} copy the
current peers, so the target keeps its raised priority until the next transfer.
Reading the group again and repeating the reset would close that, at the cost
of a larger change.
Covered by {{testBootstrapDuringLeaderTransfer}} in the existing
{{TestAddRemoveOzoneManager}}, on a ring of three voters and one listener,
which also checks that every peer has the default priority afterwards. Without
the change it fails because the bootstrapped OM is missing from the Raft
configuration of the leader. With it the test passes, as do
{{TestAddRemoveOzoneManager}} (9 tests), {{TestTransferLeadershipShell}} (5
tests, OM and SCM transfers including the priority reset check),
{{TestOMHALeaderSpecificACLEnforcement}},
{{TestSCMFollowerCatchupWithContainerReport}},
{{TestDatanodeSCMNodesReconfiguration}} and {{TestRatisHelper}}, and checkstyle
is clean. With the patch both tests of the reproduction fail, because the
membership change is kept.
Found by TLA+ model checking and code review of the OM HA membership change
paths (bootstrap, decommission and leader transfer) under HDDS-15926, on commit
7fcf31294859d5d31163e016b5a9a63f21c6edd2. Checked against HDDS issues and
apache/ozone pull requests for duplicates before filing. The attached patch is
a proposal for review. Generated with Specula (Claude Opus 5.5).
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
