[
https://issues.apache.org/jira/browse/HDDS-17205?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=18129413#comment-18129413
]
Siyao Meng commented on HDDS-17205:
-----------------------------------
{{SCMClientProtocolServer#resolveTargetPeerId}} has the same string comparison
({{node.getRatisHostPortStr().equals(peer.getAddress())}}), so {{ozone admin
scm transfer}} with a node ID (for example {{-n scm2}}) can also fail to match
when SCM addresses are literals. The fix should cover both call sites.
> SCM follower cannot install a snapshot when SCM addresses are configured as
> IP literals
> ---------------------------------------------------------------------------------------
>
> Key: HDDS-17205
> URL: https://issues.apache.org/jira/browse/HDDS-17205
> Project: Apache Ozone
> Issue Type: Bug
> Reporter: Siyao Meng
> Assignee: Siyao Meng
> Priority: Major
>
> When SCM HA addresses are configured as IP literals
> ({{ozone.scm.address.<service>.<node>=[fd00:0:0:6::11]}}), a bootstrapping
> SCM cannot install a snapshot from the leader and never joins the ring. SCM
> stays in safe mode and the leader and follower log thousands of lines per
> second.
> h3. Cause
> {{SCMStateMachine#notifyInstallSnapshotFromLeader}} looks up the leader by
> comparing the leader's Ratis address string with {{getRatisHostPortStr()}} of
> each configured peer:
> {code:java}
> .filter(p -> p.getRatisHostPortStr().equals(leaderAddress))
> ...
> Preconditions.checkState(leaderDetails.isPresent());
> {code}
> {{NodeDetails#getRatisHostPortStr}} uses {{getRpcAddress().getHostName()}}.
> For an address configured as a literal, that is a reverse lookup, and its
> result depends on the host doing the lookup. The leader resolves its own
> address through {{/etc/hosts}} (short host name), while the follower resolves
> the same address through DNS (FQDN). In the IPv6 compose cluster:
> ||lookup of fd00:0:0:6::11 in||result||
> |scm1 (leader, itself)|7fdcd1080e27|
> |scm2 (follower)|ozone-ha-scm1-1.ozone-ha_default|
> So the leader announces {{7fdcd1080e27:9894}}, the follower finds no matching
> peer, and {{checkState}} throws an {{IllegalStateException}} with no message.
> Ratis immediately sends the next install snapshot notification, which fails
> the same way, so this becomes a tight loop (scm1 wrote about 90 MB of log in
> 4 minutes).
> Host names in the configuration are not affected, because {{getHostName()}}
> returns the configured name without a lookup. Any literal (IPv4 or IPv6) is
> affected when reverse lookups disagree between hosts, which is common (short
> name in {{/etc/hosts}}, FQDN in DNS). IPv6 deployments without DNS names for
> every node are the most likely to configure literals.
> h3. Reproduction
> {{compose/ozone-ha}} on an IPv6-only network (HDDS-16969) with static
> addresses for scm1..3 and {{ozone.scm.address.scmservice.scmN}} set to the
> bracketed literals. Found while testing the HDDS-15763 patches (including
> HDDS-17135) on Docker Desktop.
> {noformat}
> ERROR impl.SnapshotInstallationHandler: ...: installSnapshot failed
> java.lang.IllegalStateException
> at
> com.google.common.base.Preconditions.checkState(Preconditions.java:519)
> at
> org.apache.hadoop.hdds.scm.ha.SCMStateMachine.notifyInstallSnapshotFromLeader(SCMStateMachine.java:285)
> {noformat}
> h3. Proposed fix
> Match the leader by resolved socket address (IP and port) instead of the host
> name string, and fail with a message naming the leader address when no peer
> matches. The tight retry loop after a failed notification is tracked
> separately in HDDS-17127.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
