[
https://issues.apache.org/jira/browse/CXF-9268?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=18137478#comment-18137478
]
Freeman Yue Fang commented on CXF-9268:
---------------------------------------
Thanks [~reta], agreed that the bigger HTTP/1.1 buffer in httpcore5 5.4.4 is
what exposes it, and #3508 avoids it on main for the default settings (16 KiB
HTTP/1.1 buffer). IMO it would still be better to change the SharedInputBuffer
a bit, because the hang itself comes from a lost wakeup in CXF's hc5
SharedInputBuffer: when one data block fills the buffer while a reader is
waiting, consumeContent() waits on suspendInput before it signals the reader,
and only the reader signals suspendInput. So it can happen whenever a block is
at least the reader's array plus chunkLength, whatever the httpcore5 version.
With the reporter's test:
||httpcore5||HTTP/1.1 buffer||chunkLength||result||
|5.4.3|8 KiB (default)|default (16320)|passes|
|5.4.3|8 KiB (default)|1024 / 2048|hangs|
|5.4.4|32 KiB|default|hangs (the reporter's case)|
|5.4.4|16 KiB (main)|default|passes|
|5.4.4|16 KiB (main)|4096 / 8192|hangs|
The code fix is small, the one the reporter suggested: signal the reader before
waiting on suspendInput. With it, all the cases above pass, including the 32
KiB buffer with the default chunkLength, and the hc5 unit tests and the async
transport systests pass. I'll open a PR with the fix and a SharedInputBuffer
unit test. The 16 KiB buffer setting from #3508 just stays as it is.
Best Regards
Freeman
> AsyncHTTPConduit (hc5) deadlocks in SharedInputBuffer with httpcore5 5.4.4
> (32 KiB HTTP/1.1 buffer); receive timeout does not fire
> ----------------------------------------------------------------------------------------------------------------------------------
>
> Key: CXF-9268
> URL: https://issues.apache.org/jira/browse/CXF-9268
> Project: CXF
> Issue Type: Bug
> Components: Transports
> Affects Versions: 4.2.4
> Environment: Java 21, cxf-rt-transports-http-hc5 4.2.4, httpcore5
> 5.4.4, httpclient5 5.6.5, AsyncHTTPConduit (USE_ASYNC=true), HTTP/1.1, no
> chunkLength set in HTTPClientPolicy
> Reporter: Markus Ackermann
> Assignee: Freeman Yue Fang
> Priority: Major
> Attachments: AsyncHTTPConduitLargeResponseHangTest.java,
> threaddump-httpcore5-5.4.4.txt
>
>
> h3. Problem
> After upgrading httpcore5 from 5.4.3 to 5.4.4 (pulled in by Spring Boot
> 4.1.2), SOAP calls via the hc5 {{AsyncHTTPConduit}} hang forever as soon as
> responses are larger than about 20 KB. The configured {{ReceiveTimeout}} does
> not fire. In production, all hc5 I/O dispatcher threads ended up blocked
> within minutes under load. After that, no request on any connection completed
> until the JVM was restarted. We saw calls hang for almost 2 hours.
> httpcore5 5.4.4 increased the default HTTP/1.1 buffer size from 8 KiB to 32
> KiB ("Increased the default HTTP/1.1 buffer size to 32 KiB",
> apache/httpcomponents-core#687). The I/O reactor therefore now passes data
> blocks of up to 32 KiB to {{CXFHttpAsyncResponseConsumer.consume()}}. That
> exposes a lost-wakeup in {{SharedInputBuffer}}.
> h3. Analysis
> Reader thread ({{SharedInputBuffer.read(byte[], int, int)}}, line 244): if
> the buffer is empty, it wraps the caller's array as {{waitingBuffer}} and
> calls {{waitForData()}}. That method signals {{suspendInput}} and then waits
> on {{condition}} without a timeout (line 170).
> I/O dispatcher thread ({{SharedInputBuffer.consumeContent()}}):
> # copies as much as fits into {{waitingBuffer}} (lines 84-87),
> # copies the *rest* of {{src}} into the internal buffer. {{ensureCapacity()}}
> grows that buffer to exactly the remaining size, so afterwards the buffer has
> no free space left (line 91),
> # because of {{!buffer().hasRemaining() && !endOfStream}}, it then calls
> {{suspendInput.await()}} (line 101),
> # and only *after* that wait does it call {{condition.signalAll()}} (line
> 106).
> The reader is waiting on {{condition}} and would see that {{waitingBuffer}}
> has been filled, but it is never woken up. The I/O thread waits on
> {{suspendInput}}, which only the reader would signal. Both threads wait
> forever.
> This deadlock occurs whenever a single {{src}} block is at least
> {{waitingBuffer.remaining() + chunkLength}}. With the default {{chunkLength}}
> of 16320 and the usual read buffers of a few KiB (for example 4096 bytes in
> {{IOUtils.copyAtLeast()}} via {{WireTapIn}}, or Woodstox's input buffer),
> that threshold is about 20 KiB:
> * With 8 KiB reactor blocks (httpcore5 <= 5.4.3), the threshold was never
> reached.
> * With 32 KiB blocks (httpcore5 5.4.4), it is reached as soon as a response
> is large enough.
> The receive timeout cannot fire either: it is enforced by the I/O reactor,
> and the reader in {{AsyncWrappedOutputStream.getHttpResponse()}} uses
> {{Object.wait()}} without a timeout. Since the I/O dispatcher thread is
> shared by all connections assigned to it, one stuck exchange blocks every
> other connection on that dispatcher.
> h3. Thread dump (excerpt)
> {code}
> // taken ~10 s into a hanging run of the attached test
> (cxf-rt-transports-http-hc5 4.2.4, httpcore5 5.4.4, httpclient5 5.6.5, JDK 21)
> // I/O dispatch thread blocked in SharedInputBuffer.consumeContent
> "httpclient-dispatch-2" daemon
> java.lang.Thread.State: WAITING (parking)
> at
> org.apache.cxf.transport.http.asyncclient.hc5.SharedInputBuffer.consumeContent(SharedInputBuffer.java:101)
> at
> org.apache.cxf.transport.http.asyncclient.hc5.CXFHttpAsyncResponseConsumer.consume(CXFHttpAsyncResponseConsumer.java:81)
> at
> org.apache.hc.client5.http.impl.async.HttpAsyncMainClientExec$1.consume(HttpAsyncMainClientExec.java:275)
> at
> org.apache.hc.core5.http.impl.nio.ClientHttp1StreamHandler.consumeData(ClientHttp1StreamHandler.java:270)
> at
> org.apache.hc.core5.http.impl.nio.ClientHttp1StreamDuplexer.consumeData(ClientHttp1StreamDuplexer.java:364)
> at
> org.apache.hc.core5.http.impl.nio.AbstractHttp1StreamDuplexer.onInput(AbstractHttp1StreamDuplexer.java:333)
> at
> org.apache.hc.core5.http.impl.nio.AbstractHttp1IOEventHandler.inputReady(AbstractHttp1IOEventHandler.java:64)
> at
> org.apache.hc.core5.http.impl.nio.ClientHttp1IOEventHandler.inputReady(ClientHttp1IOEventHandler.java:41)
> at
> org.apache.hc.core5.reactor.InternalDataChannel.onIOEvent(InternalDataChannel.java:139)
> at
> org.apache.hc.core5.reactor.InternalChannel.handleIOEvent(InternalChannel.java:51)
> ...
> // Calling thread blocked in SharedInputBuffer.waitForData
> "test-caller" daemon
> java.lang.Thread.State: WAITING (parking)
> at
> org.apache.cxf.transport.http.asyncclient.hc5.SharedInputBuffer.waitForData(SharedInputBuffer.java:170)
> at
> org.apache.cxf.transport.http.asyncclient.hc5.SharedInputBuffer.read(SharedInputBuffer.java:244)
> at
> org.apache.cxf.transport.http.asyncclient.hc5.AsyncHTTPConduit$AsyncWrappedOutputStream$6.read(AsyncHTTPConduit.java:734)
> at com.ctc.wstx.io.BaseReader.readBytes(BaseReader.java:155)
> at com.ctc.wstx.io.UTF8Reader.loadMore(UTF8Reader.java:385)
> at com.ctc.wstx.io.UTF8Reader.read(UTF8Reader.java:116)
> at com.ctc.wstx.io.MergedReader.read(MergedReader.java:104)
> at com.ctc.wstx.io.ReaderSource.readInto(ReaderSource.java:88)
> at
> com.ctc.wstx.io.BranchingReaderSource.readInto(BranchingReaderSource.java:57)
> ...
> {code}
> In production we saw exactly matching pairs: 8 dispatcher threads at
> {{SharedInputBuffer.java:101}} and 8 reader threads at
> {{SharedInputBuffer.java:170}} (via {{read(byte[],int,int)}} at line 244).
> h3. Reproduction
> The attached {{AsyncHTTPConduitLargeResponseHangTest.java}} is a
> self-contained JUnit 5 test. It needs only CXF jaxws frontend, the http-hc5
> transport, the JDK {{HttpServer}} and JUnit 5. It starts a local HTTP server
> that returns a 300 KB SOAP response with {{Content-Length}} and calls it from
> 20 threads × 5 times through {{JaxWsProxyFactoryBean}} with
> {{USE_ASYNC=true}}.
> || httpcore5 / httpclient5 || chunkLength not set (16320) || chunkLength =
> 65536 ||
> | 5.4.3 / 5.6.4 | passes | passes |
> | 5.4.4 / 5.6.5 | *hangs* (all 20 caller threads blocked, test times out
> after 15 s; reproducible) | passes |
> h3. Workaround
> Set {{HTTPClientPolicy.setChunkLength()}} well above the httpcore5 HTTP/1.1
> buffer size, e.g. 65536. Since then, the problem has not occurred in our
> production system.
> h3. Suggested fix
> In {{SharedInputBuffer.consumeContent()}}, signal {{condition}} before
> blocking on {{suspendInput}}:
> {code:java}
> if (!buffer().hasRemaining() && !this.endOfStream) {
> this.condition.signalAll(); // wake up a reader waiting on waitingBuffer
> try {
> suspendInput.await();
> } catch (InterruptedException ex) {
> throw new IOException("Interrupted while waiting buffer to be drained
> ");
> }
> }
> this.condition.signalAll();
> {code}
> This fix is untested. Independently of that, it may be worth not blocking the
> I/O dispatcher thread at all and using {{CapacityChannel}} for flow control
> instead ({{updateCapacity()}} currently always grants {{Integer.MAX_VALUE}}).
> Attachments:
> - AsyncHTTPConduitLargeResponseHangTest.java
> - threaddump-httpcore5-5.4.4.txt (full dump from the reproduction)
--
This message was sent by Atlassian Jira
(v10.3.26#10030026)