Thanks for the reply, Daniel. This was an HTTP 1.1 connection. The code snippet for building the URL and configuring the HttpClient was included in my previous email for reference.
I can't make any claims about what was happening with packet transmissions at the network level. It's entirely possible that the destination maintained the TCP connection over an extended period without fulfilling its obligation to send the body. But there must be a way to guarantee completion for such scenarios. This was the function of socket timeouts in the world of synchronous network APIs. In this new asynchronous world, it looks like that might have been forgotten. The alternative timeout code you mentioned addresses the problem in a narrow sense, but it results in a resource leak. If the HTTP session is abandoned at the application level via CompletableFuture timed wait, there is still a footprint within HttpClient. Over time, socket handles, callback objects, and other resources can accumulate with no visibility to application code. ________________________________ From: Daniel Fuchs <daniel.fu...@oracle.com> Sent: Wednesday, October 6, 2021 2:42 AM To: Elliot Barlas <elliot.bar...@logmein.com>; net-dev@openjdk.java.net <net-dev@openjdk.java.net> Subject: Re: HttpClient Send Method Guaranteed Completion Hi Eliot, Unless the server keeps the connection open, and fails to send all the body bytes it has promised to send, the send operation should eventually terminate. The behavior you describe looks indeed like a bug. Could you please log a ticket with: https://urldefense.com/v3/__https://bugreport.java.com/bugreport/__;!!OA8L0MA-!sPeWGu915xnLugfAz60R8t6hJfzTFK_1ZMX7cI0b-SgYHEDvWjnOSVCVmjsgoCLBZNA$ That said - there is very little information here to work with. It would be helpful to understand with which version of the protocol this happened: HTTP or HTTPS? Version 1.1 or Version 2? Was it with an upgrade request (h2c)? If you manage to reproduce, it would be helpful if HTTP traces could be enabled - I'd suggest: `-Djdk.httpclient.HttpClient.log=headers,requests,errors` (see https://urldefense.com/v3/__https://docs.oracle.com/en/java/javase/17/core/java-networking.html__;!!OA8L0MA-!sPeWGu915xnLugfAz60R8t6hJfzTFK_1ZMX7cI0b-SgYHEDvWjnOSVCVmjsgmoTmRXo$ ) Another issue here is that the request timeout set through `HttpRequestBuilder::timeout` only runs until the response headers are received. The reception of the body bytes is not covered by this timeout. This is not well documented, and we should probably do a better job there. We have an enhancement request related to this: https://urldefense.com/v3/__https://bugs.openjdk.java.net/browse/JDK-8258397__;!!OA8L0MA-!sPeWGu915xnLugfAz60R8t6hJfzTFK_1ZMX7cI0b-SgYHEDvWjnOSVCVmjsg46sP2Vg$ However - setting up a global timeout for the request is already possible: If you wish to set a timeout for the reception of the body bytes you can do so by making use of the timeout facility provided by CompletableFuture: https://urldefense.com/v3/__https://docs.oracle.com/en/java/javase/17/docs/api/java.base/java/util/concurrent/CompletableFuture.html*get(long,java.util.concurrent.TimeUnit)__;Iw!!OA8L0MA-!sPeWGu915xnLugfAz60R8t6hJfzTFK_1ZMX7cI0b-SgYHEDvWjnOSVCVmjsgABT3nlI$ Instead of: ``` String html = httpClient.send(request, HttpResponse.BodyHandlers.ofString()).body(); ``` Use: ``` long timeout = 5000; // ms String html; CompletableFuture<HttpResponse<String>> cf; try { cf = httpClient.sendAsync(request, HttpResponse.BodyHandlers.ofString()); html = cf.get(timeout, TimedUnit.MILLISECONDS) .body(); } catch (TimeoutException x) { // timeout: cancel request to free up // any related resources cf.cancel(); } catch (CancellationException x) { // cancelled } catch (InterruptedException x) { // interrupted } catch (ExecutionException x) { // IO etc.. // unwrap to get exception and rethrow... } ``` This offers the possibility to set up a global timeout You can also cancel the request asynchronously at any time by calling cf.cancel() (since Java 15 IIRC). best regards, -- daniel On 05/10/2021 23:05, Elliot Barlas wrote: > Hello net-dev! > > I'm emailing about a surprising observation in the HttpClient send > method related to guaranteed completion. > > Despite explicit timeout configurations, a call to send can block > indefinitely. The stacktrace below was obtained from a thread dump on a > running OpenJDK 16.0.2 JVM. The thread was stuck in that state for over > a week. An application restart was required to recover. The related Java > source code is also included below. > > My sense is that this is due to a missed notification from the async I/O > subsystem. Unfortunately, I'm not an expert on HttpClient internals, and > the asynchronous nature of the code makes it challenging to debug. > > Is this the intended behavior? That is, are there scenarios in which > send should never return? Did I miss a socket timeout option? > > ----- > > HttpClient httpClient = HttpClient.newBuilder() > .version(HttpClient.Version.HTTP_1_1) > .connectTimeout(Duration.ofSeconds(5)) > .build(); > ... > HttpRequest request = HttpRequest.newBuilder() > .timeout(Duration.ofSeconds(5)) > .uri(URI.create("http://"; + ip + ":" + port + "/frontend")) > .build(); > ... > String html = httpClient.send(request, > HttpResponse.BodyHandlers.ofString()).body(); > > ----- > > "pool-1-thread-70" #164 prio=5 os_prio=0 cpu=100038.29ms > elapsed=1891832.58s tid=0x00007fe0211d4050 nid=0x29e waiting on > condition [0x00007fdf025ac000] > java.lang.Thread.State: WAITING (parking) > at jdk.internal.misc.Unsafe.park(java.base@16.0.2/Native Method) > - parking to wait for <0x00000000e7fc9be0> (a > java.util.concurrent.CompletableFuture$Signaller) > at > java.util.concurrent.locks.LockSupport.park(java.base@16.0.2/LockSupport.java:211) > at > java.util.concurrent.CompletableFuture$Signaller.block(java.base@16.0.2/CompletableFuture.java:1860) > at > java.util.concurrent.ForkJoinPool.managedBlock(java.base@16.0.2/ForkJoinPool.java:3137) > at > java.util.concurrent.CompletableFuture.waitingGet(java.base@16.0.2/CompletableFuture.java:1894) > at > java.util.concurrent.CompletableFuture.get(java.base@16.0.2/CompletableFuture.java:2068) > at > jdk.internal.net.http.HttpClientImpl.send(java.net.http@16.0.2/HttpClientImpl.java:535) > at > jdk.internal.net.http.HttpClientFacade.send(java.net.http@16.0.2/HttpClientFacade.java:119) > at > com.logmein.haproxy.ProxyFrontendConnectionCounter.countCurrentConnections(ProxyFrontendConnectionCounter.java:89) > at > com.logmein.haproxy.ProxyFrontendConnectionCounter.doRun(ProxyFrontendConnectionCounter.java:74) > at > com.logmein.haproxy.ProxyFrontendConnectionCounter.run(ProxyFrontendConnectionCounter.java:59) > at > com.logmein.haproxy.ProxyFrontendConnectionCounter$$Lambda$712/0x00000008010a0a88.run(Unknown > Source) > at > java.util.concurrent.Executors$RunnableAdapter.call(java.base@16.0.2/Executors.java:515) > at > java.util.concurrent.FutureTask.runAndReset(java.base@16.0.2/FutureTask.java:305) > at > java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(java.base@16.0.2/ScheduledThreadPoolExecutor.java:305) > at > java.util.concurrent.ThreadPoolExecutor.runWorker(java.base@16.0.2/ThreadPoolExecutor.java:1130) > at > java.util.concurrent.ThreadPoolExecutor$Worker.run(java.base@16.0.2/ThreadPoolExecutor.java:630) > at java.lang.Thread.run(java.base@16.0.2/Thread.java:831) > > > Elliot Barlas > elliot.bar...@logmein.com
