[
https://issues.apache.org/jira/browse/HDDS-17577?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Siyao Meng updated HDDS-17577:
------------------------------
Affects Version/s: 2.2.1
2.1.2
2.1.0
2.1.1
2.2.0
> ozone admin om transfer -r can choose a listener OM as the target and fail
> with "not in FOLLOWER role"
> ------------------------------------------------------------------------------------------------------
>
> Key: HDDS-17577
> URL: https://issues.apache.org/jira/browse/HDDS-17577
> Project: Apache Ozone
> Issue Type: Bug
> Affects Versions: 2.1.0, 2.2.0, 2.1.1, 2.1.2, 2.2.1
> Reporter: Siyao Meng
> Priority: Minor
> Attachments: CR-4-om-transfer-without-target-skips-listeners.patch,
> TestBugCR4TransferWithoutTargetPicksListener.java
>
>
> h3. Mechanism
> For a transfer without a target ({{ozone admin om transfer -r}}, which sends
> {{transferLeadership("")}}), {{OzoneManager.transferLeadership}} takes the
> first peer of {{division.getGroup().getPeers()}} that is not the leader.
> Ratis builds that group from the voters and then the listeners, but
> {{RaftGroup}} keeps the peers in a {{HashMap}} keyed by peer id, so the order
> depends on the node ids (their hashes and the number of peers), not on the
> role, and a listener can come first. {{RatisHelper.transferRatisLeadership}}
> then rejects the listener with "Target ... not in FOLLOWER role". Nothing
> tries another peer.
> h3. Trigger
> # OM HA with voters {{omNode-1}}, {{omNode-2}}, {{omNode-3}} and a listener
> {{omNode-bootstrap-listener-1}} added with the normal bootstrap.
> # {{omNode-3}} is the leader. For these ids the group order is {{[omNode-3,
> omNode-bootstrap-listener-1, omNode-1, omNode-2]}}.
> # {{ozone admin om transfer -r}} (here the same call through the client API).
> Whether a cluster is affected depends on its node ids and on which voter
> leads.
> h3. Impact
> * The admin command fails every time while that leader leads, although two
> voting followers are eligible. Leadership does not move.
> * The failure happens before any configuration change, and a transfer with an
> explicit target works.
> h3. Reproduction
> PASS on unmodified source.
> [^TestBugCR4TransferWithoutTargetPicksListener.java]
> ({{hadoop-ozone/integration-test}}) starts three OMs, bootstraps the
> listener, moves leadership to the voter that comes first in the group's
> order, and asserts that {{transferLeadership("")}} fails with "Target
> omNode-bootstrap-listener-1 not in FOLLOWER role" and that the leader is
> unchanged. A passing test means the defect is present.
> h3. Patch
> [^CR-4-om-transfer-without-target-skips-listeners.patch], against
> 424482eb9cfc4b835e1bf73bcdaa9e7d70df59f0. It also applies to master at
> fef9cde752a.
> The empty target branch now only considers peers whose startup role is
> {{FOLLOWER}}, the same criterion {{RatisHelper.transferRatisLeadership}}
> checks on the same group. With no voting follower besides the leader, the
> transfer now fails with the existing "Cannot find a new leader to transfer
> leadership." instead of the listener error (argued from code, not run).
> Covered by an extension of {{testBootstrapListenerOM}} in the existing
> {{TestAddRemoveOzoneManager}}, which already runs three voters and a
> listener: it leads from the first voter in the group's order, asserts as a
> precondition that the next peer in that order is the listener (so the test
> cannot pass silently if the ids or the number of peers change the order),
> transfers without a target and asserts that a voter became leader. Without
> the change it fails with the "not in FOLLOWER role" error. With it
> {{TestAddRemoveOzoneManager}} and {{TestTransferLeadershipShell}} (13 tests)
> pass, checkstyle is clean. With the patch the reproduction fails because the
> transfer succeeds.
> Found by code review of the OM and SCM Ratis state machine integration
> (leader readiness, snapshot install and log failure handling), as part of the
> TLA+ verification effort under HDDS-15926, on commit
> 424482eb9cfc4b835e1bf73bcdaa9e7d70df59f0. Checked against HDDS issues and
> apache/ozone pull requests for duplicates before filing. The attached patch
> is a proposal for review. Generated with Specula (Claude Opus 5.5).
--
This message was sent by Atlassian Jira
(v10.3.26#10030026)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
