Aurélien Pupier created CAMEL-26696:
---------------------------------------
Summary: camel-cxf: async producer may never complete the exchange
when the connection fails
Key: CAMEL-26696
URL: https://issues.apache.org/jira/browse/CAMEL-26696
Project: Camel
Issue Type: Bug
Components: camel-cxf
Affects Versions: 4.24.0
Reporter: Aurélien Pupier
When a cxf:// producer (default, non-synchronous) calls an endpoint where the
connection fails, such as a closed port, the Camel exchange sometimes never
completes. The calling thread then waits on the
DefaultAsyncProcessorAwaitManager latch indefinitely. A route calling a SOAP
service that is down could hang instead of failing with an exception.
This was observed as a flaky test, CxfProducerTest.testInvokingAWrongServer
(CAMEL-26600). It hits the 30s @Timeout even after connection and receive
timeouts were configured on the HTTP conduit (PR #29219). Thread dump of the
stuck test thread:
java.util.concurrent.TimeoutException: testInvokingAWrongServer() timed out
after 30 seconds
Caused by: ...ExecutionTimeoutException: Execution timed out in thread
junit-timeout-thread-10
at
java.base/java.util.concurrent.CountDownLatch.await(CountDownLatch.java:230)
at
org.apache.camel.impl.engine.DefaultAsyncProcessorAwaitManager.await(DefaultAsyncProcessorAwaitManager.java:112)
at
org.apache.camel.impl.engine.DefaultAsyncProcessorAwaitManager.process(DefaultAsyncProcessorAwaitManager.java:89)
at
org.apache.camel.impl.engine.SharedCamelInternalProcessor.process(SharedCamelInternalProcessor.java:86)
at
org.apache.camel.support.cache.DefaultProducerCache.send(DefaultProducerCache.java:177)
at
org.apache.camel.impl.engine.DefaultProducerTemplate.request(DefaultProducerTemplate.java:372)
at
org.apache.camel.component.cxf.jaxws.CxfProducerTest.sendSimpleMessageWithPayloadMessage(CxfProducerTest.java:276)
at
org.apache.camel.component.cxf.jaxws.CxfProducerTest.testInvokingAWrongServer(CxfProducerTest.java:201)
The test thread is parked in the await manager. So the exchange went through
CxfProducer.process(Exchange, AsyncCallback), which returned false, and
AsyncCallback.done(...) was never called.
Analysis so far (not confirmed)
The async path completes the exchange only through
CxfClientCallback.handleResponse or handleException calling
camelAsyncCallback.done(false). Possible causes:
CXF's HTTPConduit or async client never invokes ClientCallback.handleException
for some connection failures. This might be a CXF bug, or a missing timeout or
executor configuration.
handleException is invoked, but code before camelAsyncCallback.done(false) in
the finally block throws, so the callback is skipped. Candidates in
CxfClientCallback:
endpoint.getCookieHandler().storeCookies(...) (only IOException is caught).
endpoint.getCxfBinding().populateExchangeFromCxfResponse(...).
conduitSelector.complete(cxfExchange).
Wrapping the Camel callback in its own try/finally would make this robust.
A race or thread issue in the CXF async client when the failure happens very
quickly.
The flakiness increased around 2 Nov. The camel-cxf changes of that day
(CAMEL-26143, CAMEL-26144) don't touch the async callback, so they don't seem
to be the cause. A CXF or JDK change, or the CI environment, may be involved.
Current workaround
PR #29339 sets synchronous=true on the "wrong server" endpoint of
CxfProducerTest. The test is stable, but the async error path is no longer
covered by any test.
Expected behavior
When the connection to the target fails, the exchange completes with an
exception. The error is reported via the callback, and the thread never waits
indefinitely.
Suggested next steps
Reproduce outside the flaky test, for example by looping a producer call to a
closed port with the default async mode.
Log or debug whether CxfClientCallback.handleException is reached and whether
done() is called.
Make CxfClientCallback always call camelAsyncCallback.done(false), even if the
preceding steps throw.
If the cause is in CXF, report it upstream or add a safe configuration or
timeout.
Add a test covering the async path with a failing connection, then remove the
synchronous=true workaround from CxfProducerTest.
Drafted by IBM Bob on behalf of apupier.
--
This message was sent by Atlassian Jira
(v10.3.26#10030026)
