Siyao Meng created HDDS-17437:
---------------------------------
Summary: XceiverServerDomainSocket does not release the xceiver
count of a connection it rejects at the limit
Key: HDDS-17437
URL: https://issues.apache.org/jira/browse/HDDS-17437
Project: Apache Ozone
Issue Type: Bug
Reporter: Siyao Meng
Attachments:
MC-1-release-xceiver-count-of-rejected-domain-socket-connection.patch
h3. Mechanism
{{XceiverServerDomainSocket.run}} increments {{xceriverCount}} right after
{{accept()}} and then compares it with {{maxXceiverCount}}. The only decrement
is in {{XceiverServerDomainSocket.closePeer}}, which the {{Receiver}} of an
accepted connection calls when it ends. When the limit check fails, the
{{IOException}} branch of the loop closes the peer but does not decrement, so
the rejected connection keeps its unit of the count. The same holds for every
other exit between the increment and {{daemon.start()}}: an {{IOException}}
from setting the peer timeouts or from {{Receiver.create}}, an
{{OutOfMemoryError}} from starting the thread, and the generic {{Throwable}}
branch. Nothing else adjusts the counter, so the units stay counted for the
life of the server instance.
The branch comment ("usually when the xceiver count limit is hit") shows the
branch is the intended rejection path, and
{{TestXceiverServerDomainSocket.testMaxXceiverCount}} triggers one rejection
but does not check that the count returns afterwards.
h3. Impact
* Each such exit leaves the count one higher than the number of open
connections for the life of the server instance, so afterwards the server
accepts fewer than {{maxXceiverCount}} connections. Connections it does not
accept fall back to gRPC and reads stay correct (the fallback is from code
reading).
h3. Reproduction
No standalone reproduction is attached. The regression test in the patch covers
it: {{TestXceiverServerDomainSocket.testMaxXceiverCount}}
({{hadoop-ozone/integration-test}}, needs {{libhadoop}}) now closes every
connection after the rejection, waits until the server's {{Receiver}} threads
have ended and asserts that {{getXceiverCount()}} is 0. Without the change it
fails with {{expected: 0 but was: 1}}. Only the limit rejection was run; the
other exits are from code reading. Run on macOS with a real domain socket
server and client, not on a datanode.
h3. Patch
[^MC-1-release-xceiver-count-of-rejected-domain-socket-connection.patch],
against 69d77beff60b9106e2517c36a7a1b2cc726af568. It also applies to master at
1acf315d546.
The loop body sets a flag right after the increment and clears it once
{{daemon.start()}} has handed the connection to its {{Receiver}}, which
releases the unit in {{closePeer}} as before. A {{finally}} on the loop body
decrements the count when the flag is still set, so every exit between the
increment and the hand off releases the unit exactly once. The increment stays
in the accept thread before the limit check, so the limit itself is enforced as
before. A {{@VisibleForTesting}} getter exposes the count for the test.
With the patch {{TestXceiverServerDomainSocket}} (14 tests) and
{{TestLocalChunkInputStream}} pass, and checkstyle is clean.
Found by TLA+ model checking and code review of the datanode short-circuit read
paths under HDDS-15926, on commit 69d77beff60b9106e2517c36a7a1b2cc726af568.
Checked against HDDS issues and apache/ozone pull requests for duplicates
before filing. Related but not a duplicate: HDDS-11627 (introduced the code).
The attached patch is a proposal for review. Generated with Specula (Claude
Opus 5.5).
--
This message was sent by Atlassian Jira
(v10.3.26#10030026)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
