[
https://issues.apache.org/jira/browse/CXF-9268?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Freeman Yue Fang updated CXF-9268:
----------------------------------
Fix Version/s: 3.6.14
4.2.5
4.1.10
> AsyncHTTPConduit (hc5) deadlocks in SharedInputBuffer with httpcore5 5.4.4
> (32 KiB HTTP/1.1 buffer); receive timeout does not fire
> ----------------------------------------------------------------------------------------------------------------------------------
>
> Key: CXF-9268
> URL: https://issues.apache.org/jira/browse/CXF-9268
> Project: CXF
> Issue Type: Bug
> Components: Transports
> Affects Versions: 4.2.4
> Environment: Java 21, cxf-rt-transports-http-hc5 4.2.4, httpcore5
> 5.4.4, httpclient5 5.6.5, AsyncHTTPConduit (USE_ASYNC=true), HTTP/1.1, no
> chunkLength set in HTTPClientPolicy
> Reporter: Markus Ackermann
> Assignee: Freeman Yue Fang
> Priority: Major
> Fix For: 3.6.14, 4.2.5, 4.1.10
>
> Attachments: AsyncHTTPConduitLargeResponseHangTest.java,
> threaddump-httpcore5-5.4.4.txt
>
>
> h3. Problem
> After upgrading httpcore5 from 5.4.3 to 5.4.4 (pulled in by Spring Boot
> 4.1.2), SOAP calls via the hc5 {{AsyncHTTPConduit}} hang forever as soon as
> responses are larger than about 20 KB. The configured {{ReceiveTimeout}} does
> not fire. In production, all hc5 I/O dispatcher threads ended up blocked
> within minutes under load. After that, no request on any connection completed
> until the JVM was restarted. We saw calls hang for almost 2 hours.
> httpcore5 5.4.4 increased the default HTTP/1.1 buffer size from 8 KiB to 32
> KiB ("Increased the default HTTP/1.1 buffer size to 32 KiB",
> apache/httpcomponents-core#687). The I/O reactor therefore now passes data
> blocks of up to 32 KiB to {{CXFHttpAsyncResponseConsumer.consume()}}. That
> exposes a lost-wakeup in {{SharedInputBuffer}}.
> h3. Analysis
> Reader thread ({{SharedInputBuffer.read(byte[], int, int)}}, line 244): if
> the buffer is empty, it wraps the caller's array as {{waitingBuffer}} and
> calls {{waitForData()}}. That method signals {{suspendInput}} and then waits
> on {{condition}} without a timeout (line 170).
> I/O dispatcher thread ({{SharedInputBuffer.consumeContent()}}):
> # copies as much as fits into {{waitingBuffer}} (lines 84-87),
> # copies the *rest* of {{src}} into the internal buffer. {{ensureCapacity()}}
> grows that buffer to exactly the remaining size, so afterwards the buffer has
> no free space left (line 91),
> # because of {{!buffer().hasRemaining() && !endOfStream}}, it then calls
> {{suspendInput.await()}} (line 101),
> # and only *after* that wait does it call {{condition.signalAll()}} (line
> 106).
> The reader is waiting on {{condition}} and would see that {{waitingBuffer}}
> has been filled, but it is never woken up. The I/O thread waits on
> {{suspendInput}}, which only the reader would signal. Both threads wait
> forever.
> This deadlock occurs whenever a single {{src}} block is at least
> {{waitingBuffer.remaining() + chunkLength}}. With the default {{chunkLength}}
> of 16320 and the usual read buffers of a few KiB (for example 4096 bytes in
> {{IOUtils.copyAtLeast()}} via {{WireTapIn}}, or Woodstox's input buffer),
> that threshold is about 20 KiB:
> * With 8 KiB reactor blocks (httpcore5 <= 5.4.3), the threshold was never
> reached.
> * With 32 KiB blocks (httpcore5 5.4.4), it is reached as soon as a response
> is large enough.
> The receive timeout cannot fire either: it is enforced by the I/O reactor,
> and the reader in {{AsyncWrappedOutputStream.getHttpResponse()}} uses
> {{Object.wait()}} without a timeout. Since the I/O dispatcher thread is
> shared by all connections assigned to it, one stuck exchange blocks every
> other connection on that dispatcher.
> h3. Thread dump (excerpt)
> {code}
> // taken ~10 s into a hanging run of the attached test
> (cxf-rt-transports-http-hc5 4.2.4, httpcore5 5.4.4, httpclient5 5.6.5, JDK 21)
> // I/O dispatch thread blocked in SharedInputBuffer.consumeContent
> "httpclient-dispatch-2" daemon
> java.lang.Thread.State: WAITING (parking)
> at
> org.apache.cxf.transport.http.asyncclient.hc5.SharedInputBuffer.consumeContent(SharedInputBuffer.java:101)
> at
> org.apache.cxf.transport.http.asyncclient.hc5.CXFHttpAsyncResponseConsumer.consume(CXFHttpAsyncResponseConsumer.java:81)
> at
> org.apache.hc.client5.http.impl.async.HttpAsyncMainClientExec$1.consume(HttpAsyncMainClientExec.java:275)
> at
> org.apache.hc.core5.http.impl.nio.ClientHttp1StreamHandler.consumeData(ClientHttp1StreamHandler.java:270)
> at
> org.apache.hc.core5.http.impl.nio.ClientHttp1StreamDuplexer.consumeData(ClientHttp1StreamDuplexer.java:364)
> at
> org.apache.hc.core5.http.impl.nio.AbstractHttp1StreamDuplexer.onInput(AbstractHttp1StreamDuplexer.java:333)
> at
> org.apache.hc.core5.http.impl.nio.AbstractHttp1IOEventHandler.inputReady(AbstractHttp1IOEventHandler.java:64)
> at
> org.apache.hc.core5.http.impl.nio.ClientHttp1IOEventHandler.inputReady(ClientHttp1IOEventHandler.java:41)
> at
> org.apache.hc.core5.reactor.InternalDataChannel.onIOEvent(InternalDataChannel.java:139)
> at
> org.apache.hc.core5.reactor.InternalChannel.handleIOEvent(InternalChannel.java:51)
> ...
> // Calling thread blocked in SharedInputBuffer.waitForData
> "test-caller" daemon
> java.lang.Thread.State: WAITING (parking)
> at
> org.apache.cxf.transport.http.asyncclient.hc5.SharedInputBuffer.waitForData(SharedInputBuffer.java:170)
> at
> org.apache.cxf.transport.http.asyncclient.hc5.SharedInputBuffer.read(SharedInputBuffer.java:244)
> at
> org.apache.cxf.transport.http.asyncclient.hc5.AsyncHTTPConduit$AsyncWrappedOutputStream$6.read(AsyncHTTPConduit.java:734)
> at com.ctc.wstx.io.BaseReader.readBytes(BaseReader.java:155)
> at com.ctc.wstx.io.UTF8Reader.loadMore(UTF8Reader.java:385)
> at com.ctc.wstx.io.UTF8Reader.read(UTF8Reader.java:116)
> at com.ctc.wstx.io.MergedReader.read(MergedReader.java:104)
> at com.ctc.wstx.io.ReaderSource.readInto(ReaderSource.java:88)
> at
> com.ctc.wstx.io.BranchingReaderSource.readInto(BranchingReaderSource.java:57)
> ...
> {code}
> In production we saw exactly matching pairs: 8 dispatcher threads at
> {{SharedInputBuffer.java:101}} and 8 reader threads at
> {{SharedInputBuffer.java:170}} (via {{read(byte[],int,int)}} at line 244).
> h3. Reproduction
> The attached {{AsyncHTTPConduitLargeResponseHangTest.java}} is a
> self-contained JUnit 5 test. It needs only CXF jaxws frontend, the http-hc5
> transport, the JDK {{HttpServer}} and JUnit 5. It starts a local HTTP server
> that returns a 300 KB SOAP response with {{Content-Length}} and calls it from
> 20 threads × 5 times through {{JaxWsProxyFactoryBean}} with
> {{USE_ASYNC=true}}.
> || httpcore5 / httpclient5 || chunkLength not set (16320) || chunkLength =
> 65536 ||
> | 5.4.3 / 5.6.4 | passes | passes |
> | 5.4.4 / 5.6.5 | *hangs* (all 20 caller threads blocked, test times out
> after 15 s; reproducible) | passes |
> h3. Workaround
> Set {{HTTPClientPolicy.setChunkLength()}} well above the httpcore5 HTTP/1.1
> buffer size, e.g. 65536. Since then, the problem has not occurred in our
> production system.
> h3. Suggested fix
> In {{SharedInputBuffer.consumeContent()}}, signal {{condition}} before
> blocking on {{suspendInput}}:
> {code:java}
> if (!buffer().hasRemaining() && !this.endOfStream) {
> this.condition.signalAll(); // wake up a reader waiting on waitingBuffer
> try {
> suspendInput.await();
> } catch (InterruptedException ex) {
> throw new IOException("Interrupted while waiting buffer to be drained
> ");
> }
> }
> this.condition.signalAll();
> {code}
> This fix is untested. Independently of that, it may be worth not blocking the
> I/O dispatcher thread at all and using {{CapacityChannel}} for flow control
> instead ({{updateCapacity()}} currently always grants {{Integer.MAX_VALUE}}).
> Attachments:
> - AsyncHTTPConduitLargeResponseHangTest.java
> - threaddump-httpcore5-5.4.4.txt (full dump from the reproduction)
--
This message was sent by Atlassian Jira
(v10.3.26#10030026)