[
https://issues.apache.org/jira/browse/HDDS-17135?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Siyao Meng updated HDDS-17135:
------------------------------
Description:
Several production paths still pass a host that can be a bare IPv6 literal to
{{NetUtils.createSocketAddr(String target, int defaultPort)}}, or format it
into a URL without brackets.
In Hadoop 3.4.3, {{createSocketAddr(target, defaultPort)}} parses {{target}} as
a URI authority ({{"dummyscheme://" + target}}). For a bare literal such as
{{2001:db8::10}} the URI host is null and the call throws
{{IllegalArgumentException: Does not contain a valid host:port authority}}.
Bare literals reach these paths because SCM records a datanode's IP with
{{InetAddress.getHostAddress()}}. It also records the datanode hostname with
{{getHostName()}}, which returns the literal when there is no reverse DNS.
Likewise {{NodeDetails}} takes its host address from
{{InetSocketAddress.getHostName()}}.
h3. Affected paths
# {{NodeDetails.getRpcAddress()}} (hdds-common):
{{NetUtils.createSocketAddr(hostAddress, rpcPort)}}.
{{OMNodeDetails.getFromProtobuf}} and the OM bootstrap handler build node
details from the remote OM's host address. When that address is a bare literal,
{{OzoneManager.checkRemoteOMConfig}} and {{addOMToRatisRing}} ->
{{getRatisHostPortStr()}} throw an unchecked {{IllegalArgumentException}}.
# {{SCMHANodeDetails.loadSCMHAConfig}} (server-scm):
{{NetUtils.createSocketAddr(rpcAddrStr, ratisPort)}}.
{{validateAdvertisedHost}} and {{SCMNodeInfo}} accept an unbracketed
{{ozone.scm.address.<serviceId>.<nodeId>}} such as {{2001:db8::2}}, but this
call throws for it, so SCM HA fails to start.
# {{XceiverClientManager.getPipelineCacheKey}} and the
{{XceiverClientShortCircuit}} constructor (hdds-client):
{{NetUtils.createSocketAddr(dn.getIpAddress(), port)}}. With
{{ozone.client.read.short-circuit=true}}, every non-EC read from an IPv6
datanode fails.
# {{DataNodeMetricsCollectionTask.getJmxMetricsUrl}} (Recon):
{{String.format("%s://%s:%d/jmx", ...)}} with the datanode hostname. The URL is
malformed for a literal host, so the datanode's pending-deletion metrics are
reported as -1.
# {{DiskBalancerSubCommandUtil.getSingleNodeDiskBalancerProxy}} (cli-admin): it
treats any {{:}} in the address as "port given". So {{[2001:db8::1]}} and a
bare {{2001:db8::1}} throw instead of using the default client port. Only
{{[2001:db8::1]:19864}} works.
# {{ozone local}} (ozone-tools): {{LocalOzoneCluster.address()}} and
{{getReconEndpoint()}} build {{host + ":" + port}}, and {{OzoneLocal}} prints
the SCM and OM addresses the same way. With {{--host ::1}}, every service
address becomes {{::1:9860}}. The advertised-address validation from HDDS-16308
then rejects these values.
h3. Fix
Combine the host and port with {{HddsUtils.getHostPortString(host, port)}}, and
parse the DiskBalancer address with {{HddsUtils.getHostName}} /
{{getHostPort}}, which brackets IPv6 literals and accepts DNS names, IPv4
literals and already-bracketed input. This is the same pattern used by
HDDS-15773 and HDDS-16831.
h3. Testing
Unit tests cover:
* {{OMNodeDetails}} with an unbracketed IPv6 host address
* SCM HA configuration with unbracketed and bracketed IPv6 peer addresses
* the Recon datanode JMX URL for a DNS name and an IPv6 literal over HTTP and
HTTPS
* DiskBalancer address parsing for IPv4, bare IPv6 and bracketed IPv6, with and
without a port
* {{ozone local}} service addresses, Recon endpoint and startup summary with
{{--host ::1}}
Each new test fails without the fix.
The IPv6-only acceptance test (HDDS-16969) does not catch these because compose
resolves nodes by DNS name.
was:
Several production paths still pass a host that can be a bare IPv6 literal to
{{NetUtils.createSocketAddr(String target, int defaultPort)}}, or format it
into a URL without brackets.
In Hadoop 3.4.3, {{createSocketAddr(target, defaultPort)}} parses {{target}} as
a URI authority ({{"dummyscheme://" + target}}). For a bare literal such as
{{2001:db8::10}} the URI host is null and the call throws
{{IllegalArgumentException: Does not contain a valid host:port authority}}.
Bare literals reach these paths because SCM records a datanode's IP with
{{InetAddress.getHostAddress()}}. It also records the datanode hostname with
{{getHostName()}}, which returns the literal when there is no reverse DNS.
Likewise {{NodeDetails}} takes its host address from
{{InetSocketAddress.getHostName()}}.
h3. Affected paths
# {{NodeDetails.getRpcAddress()}} (hdds-common):
{{NetUtils.createSocketAddr(hostAddress, rpcPort)}}.
{{OMNodeDetails.getFromProtobuf}} and the OM bootstrap handler build node
details from the remote OM's host address. When that address is a bare literal,
{{OzoneManager.checkRemoteOMConfig}} and {{addOMToRatisRing}} ->
{{getRatisHostPortStr()}} throw an unchecked {{IllegalArgumentException}}.
# {{SCMHANodeDetails.loadSCMHAConfig}} (server-scm):
{{NetUtils.createSocketAddr(rpcAddrStr, ratisPort)}}.
{{validateAdvertisedHost}} and {{SCMNodeInfo}} accept an unbracketed
{{ozone.scm.address.<serviceId>.<nodeId>}} such as {{2001:db8::2}}, but this
call throws for it, so SCM HA fails to start.
# {{XceiverClientManager.getPipelineCacheKey}} and the
{{XceiverClientShortCircuit}} constructor (hdds-client):
{{NetUtils.createSocketAddr(dn.getIpAddress(), port)}}. With
{{ozone.client.read.short-circuit=true}}, every non-EC read from an IPv6
datanode fails.
# {{DataNodeMetricsCollectionTask.getJmxMetricsUrl}} (Recon):
{{String.format("%s://%s:%d/jmx", ...)}} with the datanode hostname. The URL is
malformed for a literal host, so the datanode's pending-deletion metrics are
reported as -1.
h3. Fix
Combine the host and port with {{HddsUtils.getHostPortString(host, port)}},
which brackets IPv6 literals and accepts DNS names, IPv4 literals and
already-bracketed input. This is the same pattern used by HDDS-15773 and
HDDS-16831.
h3. Testing
Unit tests cover:
* {{OMNodeDetails}} with an unbracketed IPv6 host address
* SCM HA configuration with unbracketed and bracketed IPv6 peer addresses
* the Recon datanode JMX URL for a DNS name and an IPv6 literal over HTTP and
HTTPS
The IPv6-only acceptance test (HDDS-16969) does not catch these because compose
resolves nodes by DNS name.
> Bracket IPv6 literals passed to NetUtils.createSocketAddr and in Recon
> datanode JMX URLs
> ----------------------------------------------------------------------------------------
>
> Key: HDDS-17135
> URL: https://issues.apache.org/jira/browse/HDDS-17135
> Project: Apache Ozone
> Issue Type: Bug
> Reporter: Siyao Meng
> Assignee: Siyao Meng
> Priority: Major
> Attachments: HDDS-17135.001.patch, HDDS-17135.002.patch
>
>
> Several production paths still pass a host that can be a bare IPv6 literal to
> {{NetUtils.createSocketAddr(String target, int defaultPort)}}, or format it
> into a URL without brackets.
> In Hadoop 3.4.3, {{createSocketAddr(target, defaultPort)}} parses {{target}}
> as a URI authority ({{"dummyscheme://" + target}}). For a bare literal such
> as {{2001:db8::10}} the URI host is null and the call throws
> {{IllegalArgumentException: Does not contain a valid host:port authority}}.
> Bare literals reach these paths because SCM records a datanode's IP with
> {{InetAddress.getHostAddress()}}. It also records the datanode hostname with
> {{getHostName()}}, which returns the literal when there is no reverse DNS.
> Likewise {{NodeDetails}} takes its host address from
> {{InetSocketAddress.getHostName()}}.
> h3. Affected paths
> # {{NodeDetails.getRpcAddress()}} (hdds-common):
> {{NetUtils.createSocketAddr(hostAddress, rpcPort)}}.
> {{OMNodeDetails.getFromProtobuf}} and the OM bootstrap handler build node
> details from the remote OM's host address. When that address is a bare
> literal, {{OzoneManager.checkRemoteOMConfig}} and {{addOMToRatisRing}} ->
> {{getRatisHostPortStr()}} throw an unchecked {{IllegalArgumentException}}.
> # {{SCMHANodeDetails.loadSCMHAConfig}} (server-scm):
> {{NetUtils.createSocketAddr(rpcAddrStr, ratisPort)}}.
> {{validateAdvertisedHost}} and {{SCMNodeInfo}} accept an unbracketed
> {{ozone.scm.address.<serviceId>.<nodeId>}} such as {{2001:db8::2}}, but this
> call throws for it, so SCM HA fails to start.
> # {{XceiverClientManager.getPipelineCacheKey}} and the
> {{XceiverClientShortCircuit}} constructor (hdds-client):
> {{NetUtils.createSocketAddr(dn.getIpAddress(), port)}}. With
> {{ozone.client.read.short-circuit=true}}, every non-EC read from an IPv6
> datanode fails.
> # {{DataNodeMetricsCollectionTask.getJmxMetricsUrl}} (Recon):
> {{String.format("%s://%s:%d/jmx", ...)}} with the datanode hostname. The URL
> is malformed for a literal host, so the datanode's pending-deletion metrics
> are reported as -1.
> # {{DiskBalancerSubCommandUtil.getSingleNodeDiskBalancerProxy}} (cli-admin):
> it treats any {{:}} in the address as "port given". So {{[2001:db8::1]}} and
> a bare {{2001:db8::1}} throw instead of using the default client port. Only
> {{[2001:db8::1]:19864}} works.
> # {{ozone local}} (ozone-tools): {{LocalOzoneCluster.address()}} and
> {{getReconEndpoint()}} build {{host + ":" + port}}, and {{OzoneLocal}} prints
> the SCM and OM addresses the same way. With {{--host ::1}}, every service
> address becomes {{::1:9860}}. The advertised-address validation from
> HDDS-16308 then rejects these values.
> h3. Fix
> Combine the host and port with {{HddsUtils.getHostPortString(host, port)}},
> and parse the DiskBalancer address with {{HddsUtils.getHostName}} /
> {{getHostPort}}, which brackets IPv6 literals and accepts DNS names, IPv4
> literals and already-bracketed input. This is the same pattern used by
> HDDS-15773 and HDDS-16831.
> h3. Testing
> Unit tests cover:
> * {{OMNodeDetails}} with an unbracketed IPv6 host address
> * SCM HA configuration with unbracketed and bracketed IPv6 peer addresses
> * the Recon datanode JMX URL for a DNS name and an IPv6 literal over HTTP and
> HTTPS
> * DiskBalancer address parsing for IPv4, bare IPv6 and bracketed IPv6, with
> and without a port
> * {{ozone local}} service addresses, Recon endpoint and startup summary with
> {{--host ::1}}
> Each new test fails without the fix.
> The IPv6-only acceptance test (HDDS-16969) does not catch these because
> compose resolves nodes by DNS name.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
