[
https://issues.apache.org/jira/browse/HDDS-17205?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Siyao Meng updated HDDS-17205:
------------------------------
Description:
When OM or SCM HA node addresses are configured as IP literals (for example
{{ozone.scm.address.scmservice.scm1=[fd00:0:0:6::11]}}), the Ratis peer
addresses are not the literals. They are whatever reverse DNS returns for the
literal on the host that builds them. Different hosts can get different names
for the same address (short name from {{/etc/hosts}} on the node itself, FQDN
from DNS on the others), and a name may not be usable at all. Both SCM HA and
OM HA fail to form a ring in the IPv6 compose cluster with literal addresses.
h3. Cause
{{NodeDetails}} builds the Ratis address from {{getRpcAddress().getHostName()}}:
{code:java}
public String getRatisHostPortStr() {
return HddsUtils.getHostPortString(getHostName(), ratisPort);
}
{code}
For an {{InetSocketAddress}} created from a literal, {{getHostName()}} does a
reverse lookup and caches the result in the address, so later
{{getHostString()}} calls return the looked up name too. The same lookup
happens earlier in {{NodeDetails}}' constructor ({{this.hostAddress =
rpcAddress.getHostName()}}), the "Found matching ... address" log lines in
{{SCMHANodeDetails}} and {{OMHANodeDetails}}, and the HTTP address helpers in
{{OMHANodeDetails}} / {{OMNodeDetails}}. Host names in the configuration are
not affected, because {{getHostName()}} returns the configured name without a
lookup.
Lookups of the same address in the compose cluster:
||lookup of fd00:0:0:6::11 in||result||
|scm1 (itself)|7fdcd1080e27|
|scm2|ozone-ha-scm1-1.ozone-ha_default|
h3. Symptoms
*SCM HA:* the leader announces itself as {{7fdcd1080e27:9894}}.
{{SCMStateMachine#notifyInstallSnapshotFromLeader}} looks the leader up by
comparing that string with {{getRatisHostPortStr()}} of each peer
({{ozone-ha-scm1-1.ozone-ha_default:9894}}), finds nothing, and fails
{{Preconditions.checkState(leaderDetails.isPresent())}} with no message. Ratis
sends the next install snapshot notification immediately, so the bootstrapping
SCM loops forever (scm1 wrote about 90 MB of log in 4 minutes) and SCM stays in
safe mode. {{SCMClientProtocolServer#resolveTargetPeerId}} (leader transfer by
node ID) uses the same string comparison.
{noformat}
ERROR impl.SnapshotInstallationHandler: ...: installSnapshot failed
java.lang.IllegalStateException
at
com.google.common.base.Preconditions.checkState(Preconditions.java:519)
at
org.apache.hadoop.hdds.scm.ha.SCMStateMachine.notifyInstallSnapshotFromLeader(SCMStateMachine.java:285)
{noformat}
*OM HA:* the peers are {{om2|ozone-ha-om2-1.ozone-ha_default:9872}} and so on,
and no leader is elected. In this environment gRPC rejects the name because of
the underscore in the Docker network name ({{Invalid DNS name:
ozone-ha-om2-1.ozone-ha_default:9872}}). That part is specific to Docker, but
the general problem is that the configured literal is replaced by a name that
must then be resolved again. On a dual-stack host, that name can resolve to an
IPv4 address even though an IPv6 literal was configured.
h3. Reproduction
{{compose/ozone-ha}} on an IPv6-only network (HDDS-16969) with static addresses
for scm1..3 and om1..3, and {{ozone.scm.address.scmservice.scmN}} /
{{ozone.om.address.omservice.omN}} set to the bracketed literals. Found while
testing the HDDS-15763 patches (including HDDS-17135) against Hadoop
branch-3.4.4 on Docker Desktop.
h3. Proposed fix
Keep the configured host string (literal or name) for Ratis peer addresses and
for matching the leader, and do not call {{getHostName()}} on these addresses
before it is captured. Only changing {{getRatisHostPortStr()}} to
{{getHostString()}} is not enough: it was tried locally and still returned the
looked up name because of the earlier {{getHostName()}} calls. Matching the
leader by resolved socket address (IP and port) instead of by string would also
make the SCM lookup robust. The tight retry loop after a failed notification is
tracked separately in HDDS-17127.
was:
When SCM HA addresses are configured as IP literals
({{ozone.scm.address.<service>.<node>=[fd00:0:0:6::11]}}), a bootstrapping SCM
cannot install a snapshot from the leader and never joins the ring. SCM stays
in safe mode and the leader and follower log thousands of lines per second.
h3. Cause
{{SCMStateMachine#notifyInstallSnapshotFromLeader}} looks up the leader by
comparing the leader's Ratis address string with {{getRatisHostPortStr()}} of
each configured peer:
{code:java}
.filter(p -> p.getRatisHostPortStr().equals(leaderAddress))
...
Preconditions.checkState(leaderDetails.isPresent());
{code}
{{NodeDetails#getRatisHostPortStr}} uses {{getRpcAddress().getHostName()}}. For
an address configured as a literal, that is a reverse lookup, and its result
depends on the host doing the lookup. The leader resolves its own address
through {{/etc/hosts}} (short host name), while the follower resolves the same
address through DNS (FQDN). In the IPv6 compose cluster:
||lookup of fd00:0:0:6::11 in||result||
|scm1 (leader, itself)|7fdcd1080e27|
|scm2 (follower)|ozone-ha-scm1-1.ozone-ha_default|
So the leader announces {{7fdcd1080e27:9894}}, the follower finds no matching
peer, and {{checkState}} throws an {{IllegalStateException}} with no message.
Ratis immediately sends the next install snapshot notification, which fails the
same way, so this becomes a tight loop (scm1 wrote about 90 MB of log in 4
minutes).
Host names in the configuration are not affected, because {{getHostName()}}
returns the configured name without a lookup. Any literal (IPv4 or IPv6) is
affected when reverse lookups disagree between hosts, which is common (short
name in {{/etc/hosts}}, FQDN in DNS). IPv6 deployments without DNS names for
every node are the most likely to configure literals.
h3. Reproduction
{{compose/ozone-ha}} on an IPv6-only network (HDDS-16969) with static addresses
for scm1..3 and {{ozone.scm.address.scmservice.scmN}} set to the bracketed
literals. Found while testing the HDDS-15763 patches (including HDDS-17135) on
Docker Desktop.
{noformat}
ERROR impl.SnapshotInstallationHandler: ...: installSnapshot failed
java.lang.IllegalStateException
at
com.google.common.base.Preconditions.checkState(Preconditions.java:519)
at
org.apache.hadoop.hdds.scm.ha.SCMStateMachine.notifyInstallSnapshotFromLeader(SCMStateMachine.java:285)
{noformat}
h3. Proposed fix
Match the leader by resolved socket address (IP and port) instead of the host
name string, and fail with a message naming the leader address when no peer
matches. The tight retry loop after a failed notification is tracked separately
in HDDS-17127.
Summary: OM and SCM HA use reverse DNS names instead of configured IP
literals for Ratis peer addresses (was: SCM follower cannot install a snapshot
when SCM addresses are configured as IP literals)
> OM and SCM HA use reverse DNS names instead of configured IP literals for
> Ratis peer addresses
> ----------------------------------------------------------------------------------------------
>
> Key: HDDS-17205
> URL: https://issues.apache.org/jira/browse/HDDS-17205
> Project: Apache Ozone
> Issue Type: Bug
> Reporter: Siyao Meng
> Assignee: Siyao Meng
> Priority: Major
>
> When OM or SCM HA node addresses are configured as IP literals (for example
> {{ozone.scm.address.scmservice.scm1=[fd00:0:0:6::11]}}), the Ratis peer
> addresses are not the literals. They are whatever reverse DNS returns for the
> literal on the host that builds them. Different hosts can get different names
> for the same address (short name from {{/etc/hosts}} on the node itself, FQDN
> from DNS on the others), and a name may not be usable at all. Both SCM HA and
> OM HA fail to form a ring in the IPv6 compose cluster with literal addresses.
> h3. Cause
> {{NodeDetails}} builds the Ratis address from
> {{getRpcAddress().getHostName()}}:
> {code:java}
> public String getRatisHostPortStr() {
> return HddsUtils.getHostPortString(getHostName(), ratisPort);
> }
> {code}
> For an {{InetSocketAddress}} created from a literal, {{getHostName()}} does a
> reverse lookup and caches the result in the address, so later
> {{getHostString()}} calls return the looked up name too. The same lookup
> happens earlier in {{NodeDetails}}' constructor ({{this.hostAddress =
> rpcAddress.getHostName()}}), the "Found matching ... address" log lines in
> {{SCMHANodeDetails}} and {{OMHANodeDetails}}, and the HTTP address helpers in
> {{OMHANodeDetails}} / {{OMNodeDetails}}. Host names in the configuration are
> not affected, because {{getHostName()}} returns the configured name without a
> lookup.
> Lookups of the same address in the compose cluster:
> ||lookup of fd00:0:0:6::11 in||result||
> |scm1 (itself)|7fdcd1080e27|
> |scm2|ozone-ha-scm1-1.ozone-ha_default|
> h3. Symptoms
> *SCM HA:* the leader announces itself as {{7fdcd1080e27:9894}}.
> {{SCMStateMachine#notifyInstallSnapshotFromLeader}} looks the leader up by
> comparing that string with {{getRatisHostPortStr()}} of each peer
> ({{ozone-ha-scm1-1.ozone-ha_default:9894}}), finds nothing, and fails
> {{Preconditions.checkState(leaderDetails.isPresent())}} with no message.
> Ratis sends the next install snapshot notification immediately, so the
> bootstrapping SCM loops forever (scm1 wrote about 90 MB of log in 4 minutes)
> and SCM stays in safe mode. {{SCMClientProtocolServer#resolveTargetPeerId}}
> (leader transfer by node ID) uses the same string comparison.
> {noformat}
> ERROR impl.SnapshotInstallationHandler: ...: installSnapshot failed
> java.lang.IllegalStateException
> at
> com.google.common.base.Preconditions.checkState(Preconditions.java:519)
> at
> org.apache.hadoop.hdds.scm.ha.SCMStateMachine.notifyInstallSnapshotFromLeader(SCMStateMachine.java:285)
> {noformat}
> *OM HA:* the peers are {{om2|ozone-ha-om2-1.ozone-ha_default:9872}} and so
> on, and no leader is elected. In this environment gRPC rejects the name
> because of the underscore in the Docker network name ({{Invalid DNS name:
> ozone-ha-om2-1.ozone-ha_default:9872}}). That part is specific to Docker, but
> the general problem is that the configured literal is replaced by a name that
> must then be resolved again. On a dual-stack host, that name can resolve to
> an IPv4 address even though an IPv6 literal was configured.
> h3. Reproduction
> {{compose/ozone-ha}} on an IPv6-only network (HDDS-16969) with static
> addresses for scm1..3 and om1..3, and {{ozone.scm.address.scmservice.scmN}} /
> {{ozone.om.address.omservice.omN}} set to the bracketed literals. Found while
> testing the HDDS-15763 patches (including HDDS-17135) against Hadoop
> branch-3.4.4 on Docker Desktop.
> h3. Proposed fix
> Keep the configured host string (literal or name) for Ratis peer addresses
> and for matching the leader, and do not call {{getHostName()}} on these
> addresses before it is captured. Only changing {{getRatisHostPortStr()}} to
> {{getHostString()}} is not enough: it was tried locally and still returned
> the looked up name because of the earlier {{getHostName()}} calls. Matching
> the leader by resolved socket address (IP and port) instead of by string
> would also make the SCM lookup robust. The tight retry loop after a failed
> notification is tracked separately in HDDS-17127.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
