[
https://issues.apache.org/jira/browse/HDDS-16776?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Andrey Yarovoy reassigned HDDS-16776:
-------------------------------------
Assignee: Andrey Yarovoy
> OM and SCM re-run a failing topology script on every client request because
> failed rack resolutions are not cached
> ------------------------------------------------------------------------------------------------------------------
>
> Key: HDDS-16776
> URL: https://issues.apache.org/jira/browse/HDDS-16776
> Project: Apache Ozone
> Issue Type: Bug
> Components: Ozone Manager, SCM
> Reporter: Andrey Yarovoy
> Assignee: Andrey Yarovoy
> Priority: Major
>
> h2. Problem
>
> When \{{net.topology.node.switch.mapping.impl}} is left at its default of
> {\{org.apache.hadoop.net.ScriptBasedMapping}} and
> \{{net.topology.script.file.name}} points to a script
> that the service cannot execute -- missing file, wrong path, not executable,
> wrong working directory --
> both OM and SCM attempt to fork that script *once per client request*,
> forever. Nothing in the stack
> remembers that the previous attempt failed.
>
> On a cluster where the OM role could not reach its configured script driven
> by only a handful of distinct client hosts-- the rate comes from repetition,
> not from a large client population. Each failure logs a full stack trace at
> WARN.
>
> Each occurrence is a ~45-line WARN with this shape (line numbers elided --
> the observed cluster ran a vendor Hadoop build):
> {code}
> WARN [IPC Server handler 6 on default port
> 9862]-org.apache.hadoop.net.ScriptBasedMapping: \
> Exception running /path/to/topology.py <client-ip>
> java.io.IOException: Cannot run program "/path/to/topology.py" (in directory
> "<redacted>"): \
> error=2, No such file or directory
> at java.lang.ProcessBuilder.start(ProcessBuilder.java)
> at org.apache.hadoop.util.Shell.runCommand(Shell.java)
> ...
> at
> org.apache.hadoop.net.ScriptBasedMapping$RawScriptBasedMapping.resolve(ScriptBasedMapping.java)
> at
> org.apache.hadoop.net.CachedDNSToSwitchMapping.resolve(CachedDNSToSwitchMapping.java)
> at
> org.apache.hadoop.ozone.om.KeyManagerImpl.resolveNodeLocation(KeyManagerImpl.java)
> at org.apache.hadoop.ozone.om.KeyManagerImpl.getOtherNode(KeyManagerImpl.java)
> at
> org.apache.hadoop.ozone.om.KeyManagerImpl.getClientNode(KeyManagerImpl.java)
> at
> org.apache.hadoop.ozone.om.KeyManagerImpl.sortDatanodes(KeyManagerImpl.java)
> ...
> {code}
>
> h2. Root cause
> {\{ScriptBasedMapping}} extends Hadoop's \{{CachedDNSToSwitchMapping}}, so
> successful resolutions *are*
> cached -- but failures are not. The raw mapping returns \{{null}} for the
> whole batch when the script
> cannot be run (\{{ScriptBasedMapping.RawScriptBasedMapping.resolve}}), and
> the caching layer skips nulls:
> {code:java}
> // CachedDNSToSwitchMapping
> private void cacheResolvedHosts(List<String> uncachedHosts, List<String>
> resolvedHosts) {
> if (resolvedHosts != null) { // <-- a failure is never cached
> for (int i = 0; i < uncachedHosts.size(); i++) {
> cache.put(uncachedHosts.get(i), resolvedHosts.get(i));
> }
> }
> }
> {code}
> Ozone adds no negative caching on top, so every request repeats the fork.
> Both services resolve the
> *client* host on the request thread:
> * OM read path -- \{{KeyManagerImpl.sortDatanodes}} -> \{{getClientNode}} ->
> \{{getOtherNode}} ->
> {\{resolveNodeLocation}} (KeyManagerImpl.java:2311, 2382, 2388, 2404),
> reached by \{{lookupKey}},
> {\{getKeyInfo}}, \{{lookupFile}}, \{{getFileStatus}} and \{{listStatus}}.
> * OM write path -- \{{OMKeyRequest}} ->
> \{{KeyManagerImpl.sortDatanodesForWrite}} -> \{{getClientNode}}
> (OMKeyRequest.java:253, KeyManagerImpl.java:2323).
> * SCM -- \{{SCMBlockProtocolServer.getClientNode}} -> \{{getOtherNode}} ->
> {\{StorageContainerManager.resolveNodeLocation}}
> (SCMBlockProtocolServer.java:412, 422, 424), from both
> the \{{SortDatanodes}} RPC (:387) and \{{allocateBlock}}'s per-block loop
> (:212).
> Two things amplify the rate:
> # \{{getClientNode}} is invoked once per *distinct pipeline* within a single
> request. The sorted-order memoisation in \{{KeyManagerImpl.sortDatanodes}} is
> keyed by the pipeline's node set and does not cover client resolution, so one
> \{{listStatus}} spanning M pipelines performs M resolutions for the same
> client host.
>
> # The OM only matches the client against the 3-5 nodes of the pipeline being
> sorted. SCM, by contrast, consults \{{SCMNodeManager.getNodesByAddress}}
> first, so a client running on a datanode host never reaches the script there.
> OM has no such index, so co-located compute clients fall through to the
> script on nearly every request.
>
> * *Log flood.* ~45 lines per occurrence at ~1,500 occurrences/min is on the
> order of 65,000 WARN lines and ~6.7 MB of OM log *per minute*. Real OM events
> are effectively
> unfindable, and log retention is destroyed.
> * *A process fork per request, on the RPC handler thread.*
> \{{ProcessBuilder.start}} plus
> {\{Shell.runCommand}} on an IPC handler is a handler-occupancy and CPU cost
> paid on every read and every block allocation.
> * *Topology awareness silently stops working.* With the client unresolved,
> {\{NetworkTopologyImpl.sortByDistanceCost}} takes its \{{reader == null}}
> branch and *shuffles* the datanode list, so reads get a random order rather
> than a locality-ordered one. Writes keep pipeline order. Nothing reports that
> locality has been lost.
> * *No actionable signal from Ozone.* Ozone's own handler never fires --
> \{{ScriptBasedMapping}} swallows
> the \{{IOException}} and returns \{{null}}, so \{{getOtherNode}}'s \{{catch}}
> block is never entered ("Could not resolve client" appeared *zero* times in
> the log). The only evidence is a third-party WARN that Ozone cannot throttle
> or suppress.
> h2. Proposed fix
> Cache the *failure*. Add a short-lived negative cache in front of the
> configured \{{DNSToSwitchMapping}}: when resolution yields no location for a
> host, remember that for a configurable period and return no location for that
> host without invoking the mapping again.
> Default *15 minutes*, configurable, with 0 disabling the behaviour. This
> bounds the call rate to (distinct client hosts / expiry), which is
> independent of request rate -- in the observed case roughly
> *one call per minute* instead of ~N, three orders of magnitude lower, while
> still recovering
> automatically once the script is fixed, within one expiry period and without
> an OM or SCM restart.
> Keep the entries *per host*, not global. The raw mapping returns \{{null}}
> both for "the script is broken" and for "the script does not know this host",
> and these calls are single-host, so the two cases are indistinguishable at
> this layer; per-host entries bound the rate without letting one unknown host
> suppress resolution for every other client.
> Both services should share the implementation: \{{createDNSToSwitchMapping}}
> and \{{resolveNodeLocation}}
> are currently duplicated byte-for-byte between \{{StorageContainerManager}}
> (lines 2351-2370) and
> {\{KeyManagerImpl}} (lines 383-389, 2404-2414). Folding both into one helper
> in \{{hdds-server-framework}} removes the duplication and fixes OM and SCM
> together.
> h2. Out of scope
> * Resolving the client host from the topology the OM already holds (an
> address index equivalent to SCM's \{{getNodesByAddress}}). This would both
> avoid the script *and* restore correct locality for co-located clients, but
> only for clients that are datanodes -- a separate improvement.
> * Validating the configured script at service startup so a misconfiguration
> surfaces once, loudly, instead of per request.
> * Resolving the client once per request instead of once per pipeline.
> h2. Acceptance criteria
> # With a mapping that fails, a service resolves a given client host at most
> once per expiry period, and the WARN volume falls accordingly.
> # The failure is not cached when the expiry is configured to 0, preserving
> today's behaviour.
> # Once resolution starts succeeding, locations are resolved again within one
> expiry period with no restart.
> # Successful resolutions are unaffected -- they continue to be cached by the
> existing mechanism.
> # OM and SCM share one code path for creating the mapping and resolving a
> node location.
>
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
