[
https://issues.apache.org/jira/browse/HDDS-17205?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
ASF GitHub Bot updated HDDS-17205:
----------------------------------
Labels: pull-request-available (was: )
> OM and SCM HA use reverse DNS names instead of configured IP literals for
> Ratis peer addresses
> ----------------------------------------------------------------------------------------------
>
> Key: HDDS-17205
> URL: https://issues.apache.org/jira/browse/HDDS-17205
> Project: Apache Ozone
> Issue Type: Bug
> Reporter: Siyao Meng
> Assignee: Siyao Meng
> Priority: Major
> Labels: pull-request-available
>
> When OM or SCM HA node addresses are configured as IP literals (for example
> {{ozone.scm.address.scmservice.scm1=[fd00:0:0:6::11]}}), the Ratis peer
> addresses are not the literals. They are whatever reverse DNS returns for the
> literal on the host that builds them. Different hosts can get different names
> for the same address (short name from {{/etc/hosts}} on the node itself, FQDN
> from DNS on the others), and a name may not be usable at all. Both SCM HA and
> OM HA fail to form a ring in the IPv6 compose cluster with literal addresses.
> h3. Cause
> {{NodeDetails}} builds the Ratis address from
> {{getRpcAddress().getHostName()}}:
> {code:java}
> public String getRatisHostPortStr() {
> return HddsUtils.getHostPortString(getHostName(), ratisPort);
> }
> {code}
> For an {{InetSocketAddress}} created from a literal, {{getHostName()}} does a
> reverse lookup and caches the result in the address, so later
> {{getHostString()}} calls return the looked up name too. The same lookup
> happens earlier in {{NodeDetails}}' constructor ({{this.hostAddress =
> rpcAddress.getHostName()}}), the "Found matching ... address" log lines in
> {{SCMHANodeDetails}} and {{OMHANodeDetails}}, and the HTTP address helpers in
> {{OMHANodeDetails}} / {{OMNodeDetails}}. Host names in the configuration are
> not affected, because {{getHostName()}} returns the configured name without a
> lookup.
> Lookups of the same address in the compose cluster:
> ||lookup of fd00:0:0:6::11 in||result||
> |scm1 (itself)|7fdcd1080e27|
> |scm2|ozone-ha-scm1-1.ozone-ha_default|
> h3. Symptoms
> *SCM HA:* the leader announces itself as {{7fdcd1080e27:9894}}.
> {{SCMStateMachine#notifyInstallSnapshotFromLeader}} looks the leader up by
> comparing that string with {{getRatisHostPortStr()}} of each peer
> ({{ozone-ha-scm1-1.ozone-ha_default:9894}}), finds nothing, and fails
> {{Preconditions.checkState(leaderDetails.isPresent())}} with no message.
> Ratis sends the next install snapshot notification immediately, so the
> bootstrapping SCM loops forever (scm1 wrote about 90 MB of log in 4 minutes)
> and SCM stays in safe mode. {{SCMClientProtocolServer#resolveTargetPeerId}}
> (leader transfer by node ID) uses the same string comparison.
> {noformat}
> ERROR impl.SnapshotInstallationHandler: ...: installSnapshot failed
> java.lang.IllegalStateException
> at
> com.google.common.base.Preconditions.checkState(Preconditions.java:519)
> at
> org.apache.hadoop.hdds.scm.ha.SCMStateMachine.notifyInstallSnapshotFromLeader(SCMStateMachine.java:285)
> {noformat}
> *OM HA:* the peers are {{om2|ozone-ha-om2-1.ozone-ha_default:9872}} and so
> on, and no leader is elected. In this environment gRPC rejects the name
> because of the underscore in the Docker network name ({{Invalid DNS name:
> ozone-ha-om2-1.ozone-ha_default:9872}}). That part is specific to Docker, but
> the general problem is that the configured literal is replaced by a name that
> must then be resolved again. On a dual-stack host, that name can resolve to
> an IPv4 address even though an IPv6 literal was configured.
> h3. Reproduction
> {{compose/ozone-ha}} on an IPv6-only network (HDDS-16969) with static
> addresses for scm1..3 and om1..3, and {{ozone.scm.address.scmservice.scmN}} /
> {{ozone.om.address.omservice.omN}} set to the bracketed literals. Found while
> testing the HDDS-15763 patches (including HDDS-17135) against Hadoop
> branch-3.4.4 on Docker Desktop.
> h3. Proposed fix
> Keep the configured host string (literal or name) for Ratis peer addresses
> and for matching the leader, and do not call {{getHostName()}} on these
> addresses before it is captured. Only changing {{getRatisHostPortStr()}} to
> {{getHostString()}} is not enough: it was tried locally and still returned
> the looked up name because of the earlier {{getHostName()}} calls. Matching
> the leader by resolved socket address (IP and port) instead of by string
> would also make the SCM lookup robust. The tight retry loop after a failed
> notification is tracked separately in HDDS-17127.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
