Finalizers are deprecated for removal: https://openjdk.org/jeps/421
I don't think there's any particular requirement for the HttpClient to tolerate this sort of misuse, but it is nevertheless a change in behavior and I wanted to get to the bottom of it. On Thu, Jun 5, 2025 at 5:35 AM Oleg Kalnichevski <ol...@apache.org> wrote: > > Hi Ryan > > The reproducer stopped working for me (I use Ubuntu Linux) but this no > longer matters. I think there is a reasonable explanation as to why it > now may take two GC cycles to fully reclaim file descriptors in case of > a connection pool going out of scope without being properly shut down. > > I think the first GC cycle releases connections and their SSLSockets. > However SSLSockets with autoClose set to false no longer close out the > socket they have been layered on top of. With some JREs it takes > another GC cycle to release the underlying Socket objects and system > resources allocated by them. This also explains why the issue appears > to be JRE version specific. > > The best solution of course is to close HttpClient prior to letting it > ho out of scope. That would be preferred. Alternatively we could add > #finalize to managed connection classes > > https://github.com/ok2c/httpcomponents-client/commit/9cd760ee282f284824a6ce86621427babf6331fc > > Oleg > > > > On Wed, 2025-06-04 at 10:52 -0700, Ryan Schmitt wrote: > > I've updated the reproducer to use a BasicHttpClientResponseHandler. > > The behavior is the same as before. > > > > Unrelated question, but do the HttpClientResponseHandler-variant > > `#execute` methods support response streaming? All the > > implementations > > I've looked at buffer the response entity in its entirety. > > > > On Wed, Jun 4, 2025 at 5:47 AM Oleg Kalnichevski <ol...@apache.org> > > wrote: > > > > > > Hi Ryan > > > > > > I am not sure understand what exactly you are trying to do here? Do > > > you > > > intentionally use a deprecated method _and_ fail to correctly > > > return > > > connections back to the pool? Is this the scenario you are trying > > > to > > > test? > > > > > > Oleg > > > > > > > > > On Tue, 2025-06-03 at 14:15 -0700, Ryan Schmitt wrote: > > > > Furthermore, I can confirm that reverting ee0a10210 fixes the > > > > issue. > > > > No httpcore5 changes are required. > > > > > > > > On Tue, Jun 3, 2025 at 2:10 PM Ryan Schmitt <rschm...@apache.org> > > > > wrote: > > > > > > > > > > Here's a reproducer: > > > > > > > > > > https://github.com/rschmitt/httpcomponents-client/tree/gc-repro > > > > > > > > > > The bug is exhibited on JDK11 and JDK21, not on JDK8 or JDK21. > > > > > I > > > > > didn't test with any non-LTS releases. Here's JDK17: > > > > > > > > > > [0] ~/src/httpcomponents-client # j17 ./run-example.sh > > > > > SocketLeak > > > > > This Java version SHOULD exhibit the bug where two garbage > > > > > collections are required to clean up leaked sockets. > > > > > Leaking client; sockets should be in ESTABLISHED > > > > > Tue Jun 3 14:04:40 PDT 2025: > > > > > Tue Jun 3 14:04:41 PDT 2025: 2 (ESTABLISHED) > > > > > Tue Jun 3 14:04:42 PDT 2025: 3 (ESTABLISHED) > > > > > Tue Jun 3 14:04:44 PDT 2025: 3 (ESTABLISHED) > > > > > Running garbage collector; sockets should be in CLOSE_WAIT > > > > > Tue Jun 3 14:04:45 PDT 2025: 3 (CLOSE_WAIT) > > > > > Tue Jun 3 14:04:46 PDT 2025: 3 (CLOSE_WAIT) > > > > > Tue Jun 3 14:04:47 PDT 2025: 3 (CLOSE_WAIT) > > > > > Running garbage collector again; no sockets should be > > > > > listed > > > > > Tue Jun 3 14:04:48 PDT 2025: > > > > > Tue Jun 3 14:04:50 PDT 2025: > > > > > > > > > > And here's JDK21: > > > > > > > > > > [0] ~/src/httpcomponents-client # j21 ./run-example.sh > > > > > SocketLeak > > > > > This Java version SHOULD NOT exhibit the bug where two > > > > > garbage > > > > > collections are required to clean up leaked sockets. > > > > > Leaking client; sockets should be in ESTABLISHED > > > > > Tue Jun 3 14:06:25 PDT 2025: > > > > > Tue Jun 3 14:06:26 PDT 2025: 3 (ESTABLISHED) > > > > > Tue Jun 3 14:06:28 PDT 2025: 3 (ESTABLISHED) > > > > > Tue Jun 3 14:06:29 PDT 2025: 3 (ESTABLISHED) > > > > > Running garbage collector; no sockets should be listed > > > > > Tue Jun 3 14:06:30 PDT 2025: > > > > > Tue Jun 3 14:06:31 PDT 2025: > > > > > > > > > > I tested this on both macOS and Linux. > > > > > > > > > > On Fri, May 30, 2025 at 5:14 AM Oleg Kalnichevski > > > > > <ol...@apache.org> wrote: > > > > > > > > > > > > On Fri, 2025-05-30 at 11:30 +0200, Oleg Kalnichevski wrote: > > > > > > > On Thu, 2025-05-29 at 22:54 -0700, Ryan Schmitt wrote: > > > > > > > > It's the `autoClose` setting. It used to be `true`: > > > > > > > > > > > > > > > > https://github.com/apache/httpcomponents-client/blob/d8f702fb4d44c746bb0edf00643aa7139cb8bdf7/httpclient5/src/main/java/org/apache/hc/client5/http/ssl/SSLConnectionSocketFactory.java#L274 > > > > > > > > > > > > > > > > Now it's `false`: > > > > > > > > > > > > > > > > https://github.com/apache/httpcomponents-client/blob/ffe4a05f4faf3706a41ff660bcb3474c6b5101a3/httpclient5/src/main/java/org/apache/hc/client5/http/ssl/AbstractClientTlsStrategy.java#L208 > > > > > > > > > > > > > > > > This was deliberately changed in this commit: > > > > > > > > > > > > > > > > https://github.com/apache/httpcomponents-client/commit/ee0a102104d03a8d1cfe18e571d179c41242182c > > > > > > > > > > > > > > > > > > > > > > This change was introduced in response to HTTPCLIENT-2328 > > > > > > > [1]. > > > > > > > I will > > > > > > > review the code for accidental mistakes. Conceptually the > > > > > > > `autoClose` > > > > > > > setting should be merely a convenience. > > > > > > > > > > > > > > Oleg > > > > > > > > > > > > > > > > > > > > > [1] https://issues.apache.org/jira/browse/HTTPCLIENT-2328 > > > > > > > > > > > > > > > > > > > I think this change in core is more relevant [1]. At the > > > > > > moment I > > > > > > cannot see anything obviously wrong with it. > > > > > > > > > > > > Would you be able to share (contribute) a reproducer for the > > > > > > defect? > > > > > > > > > > > > Oleg > > > > > > > > > > > > [1] > > > > > > https://github.com/apache/httpcomponents-core/commit/1c95b2994f22b91bc7f54cb66ed84a8e94111a61 > > > > > > > > > > > > > > > > > > > > > > > > > > > > On Thu, May 29, 2025 at 6:08 PM Ryan Schmitt > > > > > > > > <rschm...@apache.org> > > > > > > > > wrote: > > > > > > > > > > > > > > > > > > I've been debugging a load test regression, which > > > > > > > > > turned > > > > > > > > > out to > > > > > > > > > be > > > > > > > > > caused by a client instance leak. The test started > > > > > > > > > failing > > > > > > > > > on > > > > > > > > > 5.4.4 > > > > > > > > > due to running out of file descriptors. What I realized > > > > > > > > > is > > > > > > > > > that > > > > > > > > > leaked > > > > > > > > > connection pools are eventually cleaned up by garbage > > > > > > > > > collection, > > > > > > > > > which causes the sockets' file descriptors to be > > > > > > > > > released. > > > > > > > > > What > > > > > > > > > has > > > > > > > > > changed in 5.4.4 is that this process now takes TWO > > > > > > > > > rounds > > > > > > > > > of > > > > > > > > > garbage > > > > > > > > > collection. This can be seen in the `lsof` output, > > > > > > > > > which > > > > > > > > > shows > > > > > > > > > where > > > > > > > > > in the TCP state machine the socket is: > > > > > > > > > > > > > > > > > > 5.2: ESTABLISHED -(gc)-> (gone) > > > > > > > > > 5.4: ESTABLISHED -(gc)-> CLOSE_WAIT -(gc)-> (gone) > > > > > > > > > > > > > > > > > > Does anyone know what might have changed that would > > > > > > > > > cause > > > > > > > > > this? > > > > > > > > > I'm > > > > > > > > > specifically asking about the synchronous client (I > > > > > > > > > haven't > > > > > > > > > tested > > > > > > > > > the > > > > > > > > > async client). > > > > > > > > > > > > > > > > --------------------------------------------------------- > > > > > > > > ---- > > > > > > > > ------ > > > > > > > > -- > > > > > > > > To unsubscribe, e-mail: dev-unsubscr...@hc.apache.org > > > > > > > > For additional commands, e-mail: dev-h...@hc.apache.org > > > > > > > > > > > > > > > > > > > > > > > > > > > > > ----------------------------------------------------------- > > > > > > > ---- > > > > > > > ------ > > > > > > > To unsubscribe, e-mail: dev-unsubscr...@hc.apache.org > > > > > > > For additional commands, e-mail: dev-h...@hc.apache.org > > > > > > > > > > > > > > > > > > > > > > > > > ------------------------------------------------------------- > > > > > > ---- > > > > > > ---- > > > > > > To unsubscribe, e-mail: dev-unsubscr...@hc.apache.org > > > > > > For additional commands, e-mail: dev-h...@hc.apache.org > > > > > > > > > > > > > > ----------------------------------------------------------------- > > > > ---- > > > > To unsubscribe, e-mail: dev-unsubscr...@hc.apache.org > > > > For additional commands, e-mail: dev-h...@hc.apache.org > > > > > > > > > > > > > ------------------------------------------------------------------- > > > -- > > > To unsubscribe, e-mail: dev-unsubscr...@hc.apache.org > > > For additional commands, e-mail: dev-h...@hc.apache.org > > > > > > > --------------------------------------------------------------------- > > To unsubscribe, e-mail: dev-unsubscr...@hc.apache.org > > For additional commands, e-mail: dev-h...@hc.apache.org > > > > > --------------------------------------------------------------------- > To unsubscribe, e-mail: dev-unsubscr...@hc.apache.org > For additional commands, e-mail: dev-h...@hc.apache.org > --------------------------------------------------------------------- To unsubscribe, e-mail: dev-unsubscr...@hc.apache.org For additional commands, e-mail: dev-h...@hc.apache.org
