[
https://issues.apache.org/jira/browse/HDDS-16841?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Siyao Meng updated HDDS-16841:
------------------------------
Attachment: TestBugMC5RepeatedBootstrapBlocksMembershipChanges.java
> A repeated OM bootstrap under the other Raft role blocks membership changes
> and leader transfer on the leader
> -------------------------------------------------------------------------------------------------------------
>
> Key: HDDS-16841
> URL: https://issues.apache.org/jira/browse/HDDS-16841
> Project: Apache Ozone
> Issue Type: Bug
> Reporter: Siyao Meng
> Priority: Minor
> Attachments: MC-5-reject-bootstrap-of-existing-ratis-member.patch,
> TestBugMC5RepeatedBootstrapBlocksMembershipChanges.java
>
>
> h3. Mechanism
> {{OzoneManagerRatisServer.addOMToRatisRing}} does not check whether the node
> is already a member of the Raft group. It takes the leader's voters and
> listeners and appends the node to the list for the role named in the
> bootstrap request. When the node is already a member under the other role,
> its id is in both lists of the {{SetConfigurationRequest}}.
> Ratis notices the duplicate too late.
> {{LeaderStateImpl.startSetConfiguration}} registers it as the pending
> configuration change ({{PendingRequests.addConfRequest}}) and only then
> builds the new configuration, which throws {{IllegalArgumentException}}
> ("Failed to initialize listeners: Found omNode-bootstrap-1 in existing
> peers"). Nothing clears the pending request. Every later {{setConfiguration}}
> on that leader fails the precondition in {{addConfRequest}} with
> {{IllegalStateException}} for as long as it stays leader. This is Ratis
> 3.3.1, the version Ozone uses. The current Ratis master has the same order
> (code reading, not run). The Ratis side is not changed here.
> h3. Trigger
> An OM that is already a voter is started with {{ozone om --bootstrap}} again
> while its own config now lists it in {{ozone.om.listener.nodes.<serviceId>}},
> for example to turn a voter into a listener without a decommission first. The
> config of the running OMs does not have to change: the checks on the joining
> OM pass and the request reaches the leader.
> The mirror case, a listener that is added again as a voter, fails in the same
> way at the Ratis server level (the unit test of the patch). It was not run on
> a mini cluster.
> A repeat under the same role does not have this effect: the id is then twice
> in one list and the request is rejected when it is built (code reading).
> h3. Impact
> Reproduced on unmodified source with real OMs in a mini cluster. On the same
> leader, in the same term:
> * A decommission of that OM ({{ozone admin om decommission}}) fails with
> {{IllegalStateException}}.
> * A leader transfer ({{ozone admin om transfer}}) fails with
> {{RaftRetryFailureException}} after about a minute of retries, because it
> first changes priorities with {{setConfiguration}}.
> * The bootstrap of any other OM goes through the same call and would fail too
> (code reading, not run).
> * The second start of the OM does not fail quickly. Its client retries up to
> {{ozone.client.failover.max.attempts}} times (default 500), and every attempt
> that reaches the leader is another failed request. A run without a limit was
> at 71 attempts after about 40 seconds when it was stopped. The reproduction
> sets the limit to 5.
> The Raft configuration is not changed and no data is lost. Client writes were
> not exercised. The state is gone once another OM is leader: in the
> reproduction the same decommission succeeds after Ratis has moved the
> leadership. The admin transfer command is itself blocked, so in practice the
> operator has to restart the leader OM (argued, the attached class moves the
> leadership through the Ratis server API).
> Affects 2.1.0 and later, where listeners were added (read from the release
> tags, not run there).
> h3. Reproduction
> [^TestBugMC5RepeatedBootstrapBlocksMembershipChanges.java] is self contained,
> goes into {{hadoop-ozone/integration-test}} (target path and Maven command
> are in the class comment) and passes on unmodified source when the defect is
> present. It starts three OMs in a mini cluster and bootstraps a fourth in the
> normal way. The fourth is stopped, its own config gets the listener key
> naming itself, and it is started again in bootstrap mode on its existing
> storage ({{OzoneManager.createOm}} with {{StartupOption.BOOTSTRAP}}). Then
> the class sends a decommission through {{OMAdminProtocol.decommission}} and
> calls {{OzoneManager.transferLeadership}}, checks that leader, term and Raft
> configuration are unchanged, asks the Ratis server to transfer the leadership
> (no configuration change is involved) and sends the same decommission to the
> new leader. The CLI itself is not run. No state is injected and nothing is
> timed. The only setting that differs from the defaults is the retry limit of
> the second start.
> h3. Patch
> [^MC-5-reject-bootstrap-of-existing-ratis-member.patch], against
> 7fcf31294859d5d31163e016b5a9a63f21c6edd2. It also applies to master at
> 1cc6423590f (not built there).
> {{addOMToRatisRing}} refuses a node that the Raft configuration already has
> as a voter or as a listener, before a request is built. The joining OM gets
> "OM <id> is already a member of the Ratis group <group>. Start it without
> --bootstrap." as its bootstrap error and does not retry.
> What changes for a repeated bootstrap:
> * Under the other role it is refused with that message instead of blocking
> the leader.
> * Under the same role it failed before as well, when the request was built,
> and the joining OM kept retrying. Now it is refused with that message (code
> reading for the old behaviour).
> * In two narrow cases a repeat that used to be answered with success and no
> change is now refused (code reading, not run). One is a member the leader's
> own peer list does not have, such as an OM that joined with {{--bootstrap
> --force}} without config on the running OMs and is started that way again.
> The other is a retried bootstrap request that reaches the leader after the
> first one has committed and before the leader has applied it. In both the
> unmodified code sent a list equal to the current configuration. The hint in
> the message covers them: the OM is a member and starts without
> {{--bootstrap}}.
> A member cannot change its role by bootstrapping again, with or without the
> patch. That still needs a decommission and a new bootstrap.
> Covered by {{testAddRejectsExistingMember}} in the existing
> {{TestOzoneManagerRatisServer}}: a group with one listener, the same id is
> added again as a voter and as a listener, and a later removal must still
> work. On unmodified source it fails with the Ratis
> {{IllegalArgumentException}} quoted above. With the patch
> {{TestOzoneManagerRatisServer}} (8 tests),
> {{TestOMAdminProtocolServerSideImpl}} (1 test) and
> {{TestAddRemoveOzoneManager}} (8 tests) pass and checkstyle is clean. With
> the patch the attached class fails at its assertion on the request in the
> Ratis log, because the leader refuses the second bootstrap with the message
> above and no request reaches Ratis.
> The patch of HDDS-16840 changes the same method and the same test class. The
> two patches do not apply together as they are. Whichever lands second needs a
> small rebase: the check goes in front of the place where the lists are built,
> the test helpers both patches add exist once, and the removal at the end of
> {{testAddRejectsExistingMember}} takes the node id. With both changes in one
> tree {{TestOzoneManagerRatisServer}} (12 tests) passes.
> Found by TLA+ model checking and code review of the OM HA membership change
> paths (bootstrap, decommission and leader transfer) under HDDS-15926, on
> commit 7fcf31294859d5d31163e016b5a9a63f21c6edd2. Checked against HDDS and
> RATIS issues and apache/ozone pull requests for duplicates before filing. No
> existing issue was found for the order in {{startSetConfiguration}} on the
> Ratis side. The attached patch is a proposal for review. Generated with
> Specula (Claude Opus 5.5).
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
