Siyao Meng created HDDS-17436:
---------------------------------
Summary: XceiverServerDomainSocket retries a failed accept() at
once and spins while the datanode is out of file descriptors
Key: HDDS-17436
URL: https://issues.apache.org/jira/browse/HDDS-17436
Project: Apache Ozone
Issue Type: Bug
Reporter: Siyao Meng
Attachments:
CR-3-pause-before-retrying-failed-domain-socket-accept.patch,
TestBugCR3AcceptFailureRetry.java
h3. Mechanism
{{XceiverServerDomainSocket.run}} catches every {{IOException}} other than a
timeout or an asynchronous close in one branch, which logs a WARN with the
stack trace, closes the peer and calls {{accept()}} again right away. The
branch was written for the rejection of a single connection at the xceiver
limit (its comment says "usually when the xceiver count limit is hit"). When
{{accept()}} itself fails, nothing changes before the next call, so a
persistent failure turns the loop into a busy spin. Only the
{{OutOfMemoryError}} branch waits before retrying. The loop follows HDFS
{{DataXceiverServer.run}}, which has no pause in that branch either. The
generic {{catch (Throwable)}} branch logs "XceiverServerDomainSocket: Exiting."
but keeps looping without a pause; it is not changed here.
h3. Trigger
# Short-circuit read is enabled and the datanode process has reached its open
file limit, for whatever reason outside this server.
# A short-circuit client connects. On Linux the connection stays queued and
each {{accept()}} fails at once with {{accept(2) error: Too many open files}}.
The loop spins for as long as both last. Run with an injected accept failure,
see Reproduction.
h3. Impact
* The accept thread uses most of a core and produces one WARN log event with a
stack trace per attempt: about 470k events per second in the Linux run
described under Reproduction, about 60k per second with the mocked failure on
macOS. This happens during an incident in which the datanode is already short
of resources, and the log volume can rotate away the earlier log lines of that
incident.
* It stops once descriptors are free again. No data is affected.
h3. Reproduction
PASS, deterministic, unmodified source. [^TestBugCR3AcceptFailureRetry.java]
({{hadoop-ozone/integration-test}}, no native library needed) starts the real
server on a mocked listening socket whose {{accept()}} always throws the EMFILE
{{SocketException}} and counts attempts and WARN events over one second: 58112
attempts and 58113 WARN events on macOS. A passing test means the defect is
present.
The real descriptor shortage was run separately on Linux (aarch64 container,
Zulu 17, RLIMIT_NOFILE 10240, a separate client process waiting in the backlog,
unmodified source): the accept thread used 82 to 84% of a core and produced
about 470k WARN events per second, counted in the test rather than written to a
log file, and both fell to zero once descriptors were released. That run was
not repeated with the patch; the patch was shown with the mocked failing
{{accept()}} on macOS. On macOS the real shortage does not produce the spin,
because the kernel takes the pending connection off the queue when no
descriptor is free, so {{accept()}} fails only once.
h3. Patch
[^CR-3-pause-before-retrying-failed-domain-socket-accept.patch], against
69d77beff60b9106e2517c36a7a1b2cc726af568. It also applies to master at
1acf315d546.
When the {{IOException}} came from {{accept()}} itself (no peer was created
yet), the loop sleeps one second before it calls {{accept()}} again, as the
{{OutOfMemoryError}} branch already does for 30 seconds. A connection that is
accepted and then rejected or fails is handled as before, without a pause. An
interrupt during the pause is ignored like in that branch, because {{stop()}}
clears the running flag before it interrupts the thread, so shutdown still ends
the loop at once. The interrupt status is not restored, since a set status
would make every later pause return at once and bring the spin back.
Covered by
{{TestXceiverServerDomainSocket.testAcceptFailureIsRetriedAfterPause}}, which
drives the loop with the same mocked failure and allows at most 2 attempts in
500 ms. Without the change it fails with 17767 attempts. With the patch
{{TestXceiverServerDomainSocket}} (15 tests) and {{TestLocalChunkInputStream}}
pass, checkstyle is clean, and the attached reproduction fails because at most
one attempt is made within its one second window.
Found by code review of the datanode short-circuit read paths, as part of the
TLA+ verification effort under HDDS-15926, on commit
69d77beff60b9106e2517c36a7a1b2cc726af568. Checked against HDDS issues and
apache/ozone pull requests for duplicates before filing. Related but not a
duplicate: HDDS-11627 (introduced the loop). The attached patch is a proposal
for review. Generated with Specula (Claude Opus 5.5).
--
This message was sent by Atlassian Jira
(v10.3.26#10030026)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
