[
https://issues.apache.org/jira/browse/HDDS-17437?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Siyao Meng updated HDDS-17437:
------------------------------
Affects Version/s: 2.3.0
> XceiverServerDomainSocket does not release the xceiver count of a connection
> it rejects at the limit
> ----------------------------------------------------------------------------------------------------
>
> Key: HDDS-17437
> URL: https://issues.apache.org/jira/browse/HDDS-17437
> Project: Apache Ozone
> Issue Type: Bug
> Affects Versions: 2.3.0
> Reporter: Siyao Meng
> Priority: Major
> Attachments:
> MC-1-release-xceiver-count-of-rejected-domain-socket-connection.patch
>
>
> h3. Mechanism
> {{XceiverServerDomainSocket.run}} increments {{xceriverCount}} right after
> {{accept()}} and then compares it with {{maxXceiverCount}}. The only
> decrement is in {{XceiverServerDomainSocket.closePeer}}, which the
> {{Receiver}} of an accepted connection calls when it ends. When the limit
> check fails, the {{IOException}} branch of the loop closes the peer but does
> not decrement, so the rejected connection keeps its unit of the count. The
> same holds for every other exit between the increment and {{daemon.start()}}:
> an {{IOException}} from setting the peer timeouts or from
> {{Receiver.create}}, an {{OutOfMemoryError}} from starting the thread, and
> the generic {{Throwable}} branch. Nothing else adjusts the counter, so the
> units stay counted for the life of the server instance.
> The branch comment ("usually when the xceiver count limit is hit") shows the
> branch is the intended rejection path, and
> {{TestXceiverServerDomainSocket.testMaxXceiverCount}} triggers one rejection
> but does not check that the count returns afterwards.
> h3. Impact
> * Each such exit leaves the count one higher than the number of open
> connections for the life of the server instance, so afterwards the server
> accepts fewer than {{maxXceiverCount}} connections. Connections it does not
> accept fall back to gRPC and reads stay correct (the fallback is from code
> reading).
> h3. Reproduction
> No standalone reproduction is attached. The regression test in the patch
> covers it: {{TestXceiverServerDomainSocket.testMaxXceiverCount}}
> ({{hadoop-ozone/integration-test}}, needs {{libhadoop}}) now closes every
> connection after the rejection, waits until the server's {{Receiver}} threads
> have ended and asserts that {{getXceiverCount()}} is 0. Without the change it
> fails with {{expected: 0 but was: 1}}. Only the limit rejection was run; the
> other exits are from code reading. Run on macOS with a real domain socket
> server and client, not on a datanode.
> h3. Patch
> [^MC-1-release-xceiver-count-of-rejected-domain-socket-connection.patch],
> against 69d77beff60b9106e2517c36a7a1b2cc726af568. It also applies to master
> at 1acf315d546.
> The loop body sets a flag right after the increment and clears it once
> {{daemon.start()}} has handed the connection to its {{Receiver}}, which
> releases the unit in {{closePeer}} as before. A {{finally}} on the loop body
> decrements the count when the flag is still set, so every exit between the
> increment and the hand off releases the unit exactly once. The increment
> stays in the accept thread before the limit check, so the limit itself is
> enforced as before. A {{@VisibleForTesting}} getter exposes the count for the
> test.
> With the patch {{TestXceiverServerDomainSocket}} (14 tests) and
> {{TestLocalChunkInputStream}} pass, and checkstyle is clean.
> Found by TLA+ model checking and code review of the datanode short-circuit
> read paths under HDDS-15926, on commit
> 69d77beff60b9106e2517c36a7a1b2cc726af568. Checked against HDDS issues and
> apache/ozone pull requests for duplicates before filing. Related but not a
> duplicate: HDDS-11627 (introduced the code). The attached patch is a proposal
> for review. Generated with Specula (Claude Opus 5.5).
--
This message was sent by Atlassian Jira
(v10.3.26#10030026)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
