Federico Mariani created CXF-9272:
-------------------------------------

             Summary: HttpClientHTTPConduit: async call to a refused connection 
sometimes never calls the ClientCallback (regression from CXF-9250)
                 Key: CXF-9272
                 URL: https://issues.apache.org/jira/browse/CXF-9272
             Project: CXF
          Issue Type: Bug
          Components: Transports
    Affects Versions: 4.2.4
            Reporter: Federico Mariani


Since 4.2.4, an asynchronous invocation through {{HttpClientHTTPConduit}} to an 
address that refuses the connection sometimes never completes. Neither 
{{ClientCallback.handleResponse}} nor {{handleException}} is called, so a 
caller waiting on the callback (or on the future) waits forever. The receive 
timeout does not help, because no response handling is ever scheduled.

We found this through Apache Camel's camel-cxf async producer, where an 
exchange hangs forever when the target SOAP service is down (CAMEL-26696, 
CAMEL-26600). The reproducer below uses plain CXF only.

*Reproducer* (JUnit 5, plain JAX-WS client, a fresh client per call, nothing 
listening on the port):

{code:java}
@WebService(targetNamespace = "http://example.org/echo";)
public interface Echo {
    String echo(String text);
}

@Test
void asyncCallToClosedPortAlwaysCallsBack() throws Exception {
    int port;
    try (ServerSocket s = new ServerSocket(0)) {
        port = s.getLocalPort(); // nothing listens on it after close
    }
    String address = "http://localhost:"; + port + "/echo";
    int calls = 2400;
    AtomicInteger stuck = new AtomicInteger();
    ExecutorService pool = Executors.newFixedThreadPool(16);
    CountDownLatch all = new CountDownLatch(calls);
    for (int i = 0; i < calls; i++) {
        pool.execute(() -> {
            try {
                JaxWsProxyFactoryBean factory = new JaxWsProxyFactoryBean();
                factory.setServiceClass(Echo.class);
                factory.setAddress(address);
                Client client = ClientProxy.getClient(factory.create());
                HTTPClientPolicy policy = new HTTPClientPolicy();
                policy.setConnectionTimeout(3000);
                policy.setReceiveTimeout(3000);
                ((HTTPConduit) client.getConduit()).setClient(policy);

                CountDownLatch callback = new CountDownLatch(1);
                client.invoke(new ClientCallback() {
                    @Override
                    public void handleResponse(Map<String, Object> ctx, 
Object[] res) {
                        super.handleResponse(ctx, res);
                        callback.countDown();
                    }

                    @Override
                    public void handleException(Map<String, Object> ctx, 
Throwable ex) {
                        super.handleException(ctx, ex);
                        callback.countDown();
                    }
                }, "echo", "hello");
                if (!callback.await(8, TimeUnit.SECONDS)) {
                    stuck.incrementAndGet();
                }
                client.destroy();
            } catch (Exception e) {
                // a synchronous failure from invoke() counts as completed
            } finally {
                all.countDown();
            }
        });
    }
    all.await();
    pool.shutdown();
    System.out.println(Version.getCurrentVersion() + ": stuck (no callback 
within 8s) = " + stuck.get());
}
{code}

*Results* (2 runs of 2400 calls per version, 16 threads, JDK 25.0.2 Temurin, 
macOS aarch64):

||CXF||calls||no callback within 8 s||
|4.2.3|4800|0|
|4.2.4|4800|22 (18 + 4)|
|4.2.5-SNAPSHOT (20261110.085644)|4800|42 (22 + 20)|

Sequential calls rarely hit it locally (0 in 3300 calls through the Camel 
producer). Concurrency widens the race window, as does a loaded CI machine.

*What the stuck exchanges look like*

We instrumented the Camel-based test, which uses the same async 
{{ClientImpl.invoke(ClientCallback, ...)}} path, with an out interceptor at 
{{Phase.SETUP}} that keeps the out message. Every stuck exchange shows the same 
state:
* {{outMessage.getContent(Exception.class)}} is 
{{java.util.concurrent.CompletionException: java.net.ConnectException}}
* {{exchange.getInMessage()}} is null
* {{exchange.get(ClientCallback.class)}} is still set, so {{ClientImpl}} never 
dispatched it

The calls that do complete report a {{Fault}} caused by {{ConnectException}} 
through {{handleException}}.

*Suspected cause* (our reading of the code; we did not instrument inside CXF)

CXF-9250 added this to the {{future.exceptionally(...)}} callback in 
{{HttpClientHTTPConduit.HttpClientWrappedOutputStream#setProtocolHeaders}}:
{code:java}
outMessage.setContent(Exception.class, ex instanceof Exception ? (Exception) ex 
: new IOException(ex));
...
close();
{code}
{{HTTPConduit.WrappedOutputStream#handleResponseOnWorkqueue}} only schedules 
the runnable that calls {{handleResponseInternal()}}, and the fault observer if 
that fails, when no exception is recorded:
{code:java}
boolean exceptionSet = outMessage.getContent(Exception.class) != null;
if (!exceptionSet) {
    ... execute(runnable) ...
}
{code}
If the {{exceptionally}} callback records the {{ConnectException}} before the 
caller thread's {{close()}} reaches {{handleResponse()}}, both {{close()}} 
calls (the caller's and the one from {{exceptionally}}) skip scheduling. 
Nothing then calls the fault observer, so the {{ClientCallback}} is never 
invoked. The raw {{CompletionException}} left on the out message of the stuck 
exchanges matches this path. The 4.2.3 to 4.2.4 difference also points at 
CXF-9250: between the {{cxf-4.2.3}} and {{cxf-4.2.4}} tags it is the only 
commit touching {{HttpClientHTTPConduit}} or {{HTTPConduit}}.

CXF-9250 also went into 4.1.9, 4.0.12 and 3.6.13. We did not test those 
versions.

Possibly related, but a different cause: CXF-9257 (the callback throws) and 
CXF-9267 (a race in {{HttpClientBodyPublisher}} hides the connect timeout). 
Both are fixed on main, and this issue still reproduces on the 4.2.5-SNAPSHOT 
above.

_Claude Code on behalf of Croway_




--
This message was sent by Atlassian Jira
(v10.3.26#10030026)

Reply via email to