Markus Ackermann created CXF-9268:
-------------------------------------
Summary: AsyncHTTPConduit (hc5) deadlocks in SharedInputBuffer
with httpcore5 5.4.4 (32 KiB HTTP/1.1 buffer); receive timeout does not fire
Key: CXF-9268
URL: https://issues.apache.org/jira/browse/CXF-9268
Project: CXF
Issue Type: Bug
Components: Transports
Affects Versions: 4.2.4
Environment: Java 21, cxf-rt-transports-http-hc5 4.2.4, httpcore5
5.4.4, httpclient5 5.6.5, AsyncHTTPConduit (USE_ASYNC=true), HTTP/1.1, no
chunkLength set in HTTPClientPolicy
Reporter: Markus Ackermann
Attachments: AsyncHTTPConduitLargeResponseHangTest.java,
threaddump-httpcore5-5.4.4.txt
h3. Problem
After upgrading httpcore5 from 5.4.3 to 5.4.4 (pulled in by Spring Boot 4.1.2),
SOAP calls via the hc5 {{AsyncHTTPConduit}} hang forever as soon as responses
are larger than about 20 KB. The configured {{ReceiveTimeout}} does not fire.
In production, all hc5 I/O dispatcher threads ended up blocked within minutes
under load. After that, no request on any connection completed until the JVM
was restarted. We saw calls hang for almost 2 hours.
httpcore5 5.4.4 increased the default HTTP/1.1 buffer size from 8 KiB to 32 KiB
("Increased the default HTTP/1.1 buffer size to 32 KiB",
apache/httpcomponents-core#687). The I/O reactor therefore now passes data
blocks of up to 32 KiB to {{CXFHttpAsyncResponseConsumer.consume()}}. That
exposes a lost-wakeup in {{SharedInputBuffer}}.
h3. Analysis
Reader thread ({{SharedInputBuffer.read(byte[], int, int)}}, line 244): if the
buffer is empty, it wraps the caller's array as {{waitingBuffer}} and calls
{{waitForData()}}. That method signals {{suspendInput}} and then waits on
{{condition}} without a timeout (line 170).
I/O dispatcher thread ({{SharedInputBuffer.consumeContent()}}):
# copies as much as fits into {{waitingBuffer}} (lines 84-87),
# copies the *rest* of {{src}} into the internal buffer. {{ensureCapacity()}}
grows that buffer to exactly the remaining size, so afterwards the buffer has
no free space left (line 91),
# because of {{!buffer().hasRemaining() && !endOfStream}}, it then calls
{{suspendInput.await()}} (line 101),
# and only *after* that wait does it call {{condition.signalAll()}} (line 106).
The reader is waiting on {{condition}} and would see that {{waitingBuffer}} has
been filled, but it is never woken up. The I/O thread waits on
{{suspendInput}}, which only the reader would signal. Both threads wait forever.
This deadlock occurs whenever a single {{src}} block is at least
{{waitingBuffer.remaining() + chunkLength}}. With the default {{chunkLength}}
of 16320 and the usual read buffers of a few KiB (for example 4096 bytes in
{{IOUtils.copyAtLeast()}} via {{WireTapIn}}, or Woodstox's input buffer), that
threshold is about 20 KiB:
* With 8 KiB reactor blocks (httpcore5 <= 5.4.3), the threshold was never
reached.
* With 32 KiB blocks (httpcore5 5.4.4), it is reached as soon as a response is
large enough.
The receive timeout cannot fire either: it is enforced by the I/O reactor, and
the reader in {{AsyncWrappedOutputStream.getHttpResponse()}} uses
{{Object.wait()}} without a timeout. Since the I/O dispatcher thread is shared
by all connections assigned to it, one stuck exchange blocks every other
connection on that dispatcher.
h3. Thread dump (excerpt)
{code}
// taken ~10 s into a hanging run of the attached test
(cxf-rt-transports-http-hc5 4.2.4, httpcore5 5.4.4, httpclient5 5.6.5, JDK 21)
// I/O dispatch thread blocked in SharedInputBuffer.consumeContent
"httpclient-dispatch-2" daemon
java.lang.Thread.State: WAITING (parking)
at
org.apache.cxf.transport.http.asyncclient.hc5.SharedInputBuffer.consumeContent(SharedInputBuffer.java:101)
at
org.apache.cxf.transport.http.asyncclient.hc5.CXFHttpAsyncResponseConsumer.consume(CXFHttpAsyncResponseConsumer.java:81)
at
org.apache.hc.client5.http.impl.async.HttpAsyncMainClientExec$1.consume(HttpAsyncMainClientExec.java:275)
at
org.apache.hc.core5.http.impl.nio.ClientHttp1StreamHandler.consumeData(ClientHttp1StreamHandler.java:270)
at
org.apache.hc.core5.http.impl.nio.ClientHttp1StreamDuplexer.consumeData(ClientHttp1StreamDuplexer.java:364)
at
org.apache.hc.core5.http.impl.nio.AbstractHttp1StreamDuplexer.onInput(AbstractHttp1StreamDuplexer.java:333)
at
org.apache.hc.core5.http.impl.nio.AbstractHttp1IOEventHandler.inputReady(AbstractHttp1IOEventHandler.java:64)
at
org.apache.hc.core5.http.impl.nio.ClientHttp1IOEventHandler.inputReady(ClientHttp1IOEventHandler.java:41)
at
org.apache.hc.core5.reactor.InternalDataChannel.onIOEvent(InternalDataChannel.java:139)
at
org.apache.hc.core5.reactor.InternalChannel.handleIOEvent(InternalChannel.java:51)
...
// Calling thread blocked in SharedInputBuffer.waitForData
"test-caller" daemon
java.lang.Thread.State: WAITING (parking)
at
org.apache.cxf.transport.http.asyncclient.hc5.SharedInputBuffer.waitForData(SharedInputBuffer.java:170)
at
org.apache.cxf.transport.http.asyncclient.hc5.SharedInputBuffer.read(SharedInputBuffer.java:244)
at
org.apache.cxf.transport.http.asyncclient.hc5.AsyncHTTPConduit$AsyncWrappedOutputStream$6.read(AsyncHTTPConduit.java:734)
at com.ctc.wstx.io.BaseReader.readBytes(BaseReader.java:155)
at com.ctc.wstx.io.UTF8Reader.loadMore(UTF8Reader.java:385)
at com.ctc.wstx.io.UTF8Reader.read(UTF8Reader.java:116)
at com.ctc.wstx.io.MergedReader.read(MergedReader.java:104)
at com.ctc.wstx.io.ReaderSource.readInto(ReaderSource.java:88)
at
com.ctc.wstx.io.BranchingReaderSource.readInto(BranchingReaderSource.java:57)
...
{code}
In production we saw exactly matching pairs: 8 dispatcher threads at
{{SharedInputBuffer.java:101}} and 8 reader threads at
{{SharedInputBuffer.java:170}} (via {{read(byte[],int,int)}} at line 244).
h3. Reproduction
The attached {{AsyncHTTPConduitLargeResponseHangTest.java}} is a self-contained
JUnit 5 test. It needs only CXF jaxws frontend, the http-hc5 transport, the JDK
{{HttpServer}} and JUnit 5. It starts a local HTTP server that returns a 300 KB
SOAP response with {{Content-Length}} and calls it from 20 threads × 5 times
through {{JaxWsProxyFactoryBean}} with {{USE_ASYNC=true}}.
|| httpcore5 / httpclient5 || chunkLength not set (16320) || chunkLength =
65536 ||
| 5.4.3 / 5.6.4 | passes | passes |
| 5.4.4 / 5.6.5 | *hangs* (all 20 caller threads blocked, test times out after
15 s; reproducible) | passes |
h3. Workaround
Set {{HTTPClientPolicy.setChunkLength()}} well above the httpcore5 HTTP/1.1
buffer size, e.g. 65536. Since then, the problem has not occurred in our
production system.
h3. Suggested fix
In {{SharedInputBuffer.consumeContent()}}, signal {{condition}} before blocking
on {{suspendInput}}:
{code:java}
if (!buffer().hasRemaining() && !this.endOfStream) {
this.condition.signalAll(); // wake up a reader waiting on waitingBuffer
try {
suspendInput.await();
} catch (InterruptedException ex) {
throw new IOException("Interrupted while waiting buffer to be drained
");
}
}
this.condition.signalAll();
{code}
This fix is untested. Independently of that, it may be worth not blocking the
I/O dispatcher thread at all and using {{CapacityChannel}} for flow control
instead ({{updateCapacity()}} currently always grants {{Integer.MAX_VALUE}}).
Attachments:
- AsyncHTTPConduitLargeResponseHangTest.java
- threaddump-httpcore5-5.4.4.txt (full dump from the reproduction)
--
This message was sent by Atlassian Jira
(v10.3.26#10030026)