Siyao Meng created HDDS-17205:
---------------------------------
Summary: SCM follower cannot install a snapshot when SCM addresses
are configured as IP literals
Key: HDDS-17205
URL: https://issues.apache.org/jira/browse/HDDS-17205
Project: Apache Ozone
Issue Type: Bug
Reporter: Siyao Meng
Assignee: Siyao Meng
When SCM HA addresses are configured as IP literals
({{ozone.scm.address.<service>.<node>=[fd00:0:0:6::11]}}), a bootstrapping SCM
cannot install a snapshot from the leader and never joins the ring. SCM stays
in safe mode and the leader and follower log thousands of lines per second.
h3. Cause
{{SCMStateMachine#notifyInstallSnapshotFromLeader}} looks up the leader by
comparing the leader's Ratis address string with {{getRatisHostPortStr()}} of
each configured peer:
{code:java}
.filter(p -> p.getRatisHostPortStr().equals(leaderAddress))
...
Preconditions.checkState(leaderDetails.isPresent());
{code}
{{NodeDetails#getRatisHostPortStr}} uses {{getRpcAddress().getHostName()}}. For
an address configured as a literal, that is a reverse lookup, and its result
depends on the host doing the lookup. The leader resolves its own address
through {{/etc/hosts}} (short host name), while the follower resolves the same
address through DNS (FQDN). In the IPv6 compose cluster:
||lookup of fd00:0:0:6::11 in||result||
|scm1 (leader, itself)|7fdcd1080e27|
|scm2 (follower)|ozone-ha-scm1-1.ozone-ha_default|
So the leader announces {{7fdcd1080e27:9894}}, the follower finds no matching
peer, and {{checkState}} throws an {{IllegalStateException}} with no message.
Ratis immediately sends the next install snapshot notification, which fails the
same way, so this becomes a tight loop (scm1 wrote about 90 MB of log in 4
minutes).
Host names in the configuration are not affected, because {{getHostName()}}
returns the configured name without a lookup. Any literal (IPv4 or IPv6) is
affected when reverse lookups disagree between hosts, which is common (short
name in {{/etc/hosts}}, FQDN in DNS). IPv6 deployments without DNS names for
every node are the most likely to configure literals.
h3. Reproduction
{{compose/ozone-ha}} on an IPv6-only network (HDDS-16969) with static addresses
for scm1..3 and {{ozone.scm.address.scmservice.scmN}} set to the bracketed
literals. Found while testing the HDDS-15763 patches (including HDDS-17135) on
Docker Desktop.
{noformat}
ERROR impl.SnapshotInstallationHandler: ...: installSnapshot failed
java.lang.IllegalStateException
at
com.google.common.base.Preconditions.checkState(Preconditions.java:519)
at
org.apache.hadoop.hdds.scm.ha.SCMStateMachine.notifyInstallSnapshotFromLeader(SCMStateMachine.java:285)
{noformat}
h3. Proposed fix
Match the leader by resolved socket address (IP and port) instead of the host
name string, and fail with a message naming the leader address when no peer
matches. The tight retry loop after a failed notification is tracked separately
in HDDS-17127.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
