[
https://issues.apache.org/jira/browse/HDDS-16836?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Siyao Meng updated HDDS-16836:
------------------------------
Attachment: CR-2-take-listener-role-in-service-list-from-raft-conf.patch
> ozone admin om roles reports LISTENER or FOLLOWER from the local config of
> the answering OM, not from the Raft configuration
> ----------------------------------------------------------------------------------------------------------------------------
>
> Key: HDDS-16836
> URL: https://issues.apache.org/jira/browse/HDDS-16836
> Project: Apache Ozone
> Issue Type: Bug
> Reporter: Siyao Meng
> Priority: Minor
> Attachments:
> CR-2-take-listener-role-in-service-list-from-raft-conf.patch,
> TestBugCR2ServiceListFromLocalConfig.java
>
>
> h3. Mechanism
> {{OzoneManager.getServiceList}} takes LISTENER or FOLLOWER for each OM from
> {{OMNodeDetails.isRatisListener}}, which is what the config of the answering
> OM says. Only the leader comes from Ratis. {{OzoneManager.getRatisRoles}}
> (the OM web UI and JMX) formats the same list. The reported role can
> therefore differ from the committed Raft configuration.
> The Raft role of a new OM is decided by the config of the new OM
> ({{BootstrapOMRequest.isListener}}), while every existing OM reads the flag
> for it from its own config. {{OzoneManager.checkRemoteOMConfig}} compares the
> address and the decommissioned nodes list of the two configs, not
> {{ozone.om.listener.nodes}}.
> Present since listener OMs were added in 2.1.0 (HDDS-11523).
> h3. Trigger
> A normal {{ozone om --bootstrap}} of an OM whose
> {{ozone.om.listener.nodes.<serviceId>}} differs from the one on the existing
> OMs. The config check passes in both directions.
> * The new OM is in the key only in its own config (run): Ratis adds a
> listener, and every existing OM reports it as FOLLOWER.
> * The new OM is in the key on the existing OMs and not in its own config
> (from the code, not run through a bootstrap; the regression test produces the
> same view by setting the flag on the leader): Ratis adds a voter, and every
> existing OM reports it as LISTENER. This is the direction in which the report
> hides a change of the quorum size.
> h3. Impact
> Reporting only. {{ozone admin om roles}}, the OM web UI, JMX and
> {{RpcClient.getOmRoleInfos}} show a role that differs from the Raft
> configuration. I found no reader of this list that makes a membership, quorum
> or failover decision from the role: the client failover proxy providers and
> the decommission and transfer commands use the config, Recon reads only the
> leader entry, the S3 gateway and the client read versions and certificates.
> The exception is the Freon OM metadata generator, a benchmark tool, which
> picks FOLLOWER entries for follower reads.
> The same difference between the local view and the Raft configuration is
> harmful where the leader builds a new Raft configuration from it. That is a
> separate defect and not part of this issue.
> h3. Reproduction
> PASS, no fault injection and no timing hooks, unmodified source.
> [^TestBugCR2ServiceListFromLocalConfig.java]
> ({{hadoop-ozone/integration-test}}) uses the real OMs of a mini cluster. The
> first test shows the listener reported as FOLLOWER by all three existing OMs
> in {{getServiceList}} and {{getRatisRoles}} while their Raft configuration
> has it as listener. The second test shows the related gap below. A passing
> test means the defect is present.
> h3. Suggested fix
> Take the role from the Raft configuration (the attached patch). The bootstrap
> config check could also compare {{ozone.om.listener.nodes}}, so that the
> difference cannot arise through a normal bootstrap; that does not replace the
> patch for a forced bootstrap or for a ring that is already in that state.
> h3. Patch
> [^CR-2-take-listener-role-in-service-list-from-raft-conf.patch], against
> 7fcf31294859d5d31163e016b5a9a63f21c6edd2. It also applies to master at
> 1cc6423590f (not built there).
> {{OzoneManager.getServiceList}} now reports an OM as LISTENER if the Raft
> configuration of the answering OM has it as a listener, for the answering OM
> and for its peers. The listeners are read through the Ratis division the
> method already uses for the leader ID, so no new exception can come out of
> the call. Nothing else in the list changes.
> Covered by {{testServiceListHasListenersOfRaftConf}} in the existing
> {{TestAddRemoveOzoneManager}}. The test gives a voter the listener flag in
> the local view of the leader and expects the list to follow the Raft
> configuration. Without the change it fails because that voter is reported as
> LISTENER. With it the nine tests of {{TestAddRemoveOzoneManager}} and
> {{TestServiceInfoProvider}} pass and checkstyle is clean. With the patch the
> first test of the reproduction fails because the new OM is reported as
> LISTENER, and the second still passes.
> h3. Related, not fixed by this patch
> The set of OMs in the list also comes from the local view ({{peerNodesMap}}).
> {{OzoneManager.addOMNodeToPeers}} logs an error and returns when the local
> config has no address for a new member, after the change is already
> committed. After {{ozone om --bootstrap --force}} against existing OMs whose
> config does not have the new OM, Ratis has a fourth voter and the existing
> OMs keep reporting three OMs (run, second test of the reproduction). Such a
> member cannot be listed completely, because each entry carries the host name
> and RPC port from the local config and the Raft configuration only has the
> Ratis address.
> Found by code review of the OM HA membership change paths (bootstrap,
> decommission and leader transfer), as part of the TLA+ verification effort
> under HDDS-15926, on commit 7fcf31294859d5d31163e016b5a9a63f21c6edd2. Checked
> against HDDS issues and apache/ozone pull requests for duplicates before
> filing. The attached patch is a proposal for review. Generated with Specula
> (Claude Opus 5.5).
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
