I think it is worth noting that GC behavior can not only differ between Java versions but can also be affected by JVM command line parameters. So explaing GC precisely as it relates to our code in an application is very difficult.
Gary On Thu, Jun 5, 2025, 08:35 Oleg Kalnichevski <ol...@apache.org> wrote: > Hi Ryan > > The reproducer stopped working for me (I use Ubuntu Linux) but this no > longer matters. I think there is a reasonable explanation as to why it > now may take two GC cycles to fully reclaim file descriptors in case of > a connection pool going out of scope without being properly shut down. > > I think the first GC cycle releases connections and their SSLSockets. > However SSLSockets with autoClose set to false no longer close out the > socket they have been layered on top of. With some JREs it takes > another GC cycle to release the underlying Socket objects and system > resources allocated by them. This also explains why the issue appears > to be JRE version specific. > > The best solution of course is to close HttpClient prior to letting it > ho out of scope. That would be preferred. Alternatively we could add > #finalize to managed connection classes > > > https://github.com/ok2c/httpcomponents-client/commit/9cd760ee282f284824a6ce86621427babf6331fc > > Oleg > > > > On Wed, 2025-06-04 at 10:52 -0700, Ryan Schmitt wrote: > > I've updated the reproducer to use a BasicHttpClientResponseHandler. > > The behavior is the same as before. > > > > Unrelated question, but do the HttpClientResponseHandler-variant > > `#execute` methods support response streaming? All the > > implementations > > I've looked at buffer the response entity in its entirety. > > > > On Wed, Jun 4, 2025 at 5:47 AM Oleg Kalnichevski <ol...@apache.org> > > wrote: > > > > > > Hi Ryan > > > > > > I am not sure understand what exactly you are trying to do here? Do > > > you > > > intentionally use a deprecated method _and_ fail to correctly > > > return > > > connections back to the pool? Is this the scenario you are trying > > > to > > > test? > > > > > > Oleg > > > > > > > > > On Tue, 2025-06-03 at 14:15 -0700, Ryan Schmitt wrote: > > > > Furthermore, I can confirm that reverting ee0a10210 fixes the > > > > issue. > > > > No httpcore5 changes are required. > > > > > > > > On Tue, Jun 3, 2025 at 2:10 PM Ryan Schmitt <rschm...@apache.org> > > > > wrote: > > > > > > > > > > Here's a reproducer: > > > > > > > > > > https://github.com/rschmitt/httpcomponents-client/tree/gc-repro > > > > > > > > > > The bug is exhibited on JDK11 and JDK21, not on JDK8 or JDK21. > > > > > I > > > > > didn't test with any non-LTS releases. Here's JDK17: > > > > > > > > > > [0] ~/src/httpcomponents-client # j17 ./run-example.sh > > > > > SocketLeak > > > > > This Java version SHOULD exhibit the bug where two garbage > > > > > collections are required to clean up leaked sockets. > > > > > Leaking client; sockets should be in ESTABLISHED > > > > > Tue Jun 3 14:04:40 PDT 2025: > > > > > Tue Jun 3 14:04:41 PDT 2025: 2 (ESTABLISHED) > > > > > Tue Jun 3 14:04:42 PDT 2025: 3 (ESTABLISHED) > > > > > Tue Jun 3 14:04:44 PDT 2025: 3 (ESTABLISHED) > > > > > Running garbage collector; sockets should be in CLOSE_WAIT > > > > > Tue Jun 3 14:04:45 PDT 2025: 3 (CLOSE_WAIT) > > > > > Tue Jun 3 14:04:46 PDT 2025: 3 (CLOSE_WAIT) > > > > > Tue Jun 3 14:04:47 PDT 2025: 3 (CLOSE_WAIT) > > > > > Running garbage collector again; no sockets should be > > > > > listed > > > > > Tue Jun 3 14:04:48 PDT 2025: > > > > > Tue Jun 3 14:04:50 PDT 2025: > > > > > > > > > > And here's JDK21: > > > > > > > > > > [0] ~/src/httpcomponents-client # j21 ./run-example.sh > > > > > SocketLeak > > > > > This Java version SHOULD NOT exhibit the bug where two > > > > > garbage > > > > > collections are required to clean up leaked sockets. > > > > > Leaking client; sockets should be in ESTABLISHED > > > > > Tue Jun 3 14:06:25 PDT 2025: > > > > > Tue Jun 3 14:06:26 PDT 2025: 3 (ESTABLISHED) > > > > > Tue Jun 3 14:06:28 PDT 2025: 3 (ESTABLISHED) > > > > > Tue Jun 3 14:06:29 PDT 2025: 3 (ESTABLISHED) > > > > > Running garbage collector; no sockets should be listed > > > > > Tue Jun 3 14:06:30 PDT 2025: > > > > > Tue Jun 3 14:06:31 PDT 2025: > > > > > > > > > > I tested this on both macOS and Linux. > > > > > > > > > > On Fri, May 30, 2025 at 5:14 AM Oleg Kalnichevski > > > > > <ol...@apache.org> wrote: > > > > > > > > > > > > On Fri, 2025-05-30 at 11:30 +0200, Oleg Kalnichevski wrote: > > > > > > > On Thu, 2025-05-29 at 22:54 -0700, Ryan Schmitt wrote: > > > > > > > > It's the `autoClose` setting. It used to be `true`: > > > > > > > > > > > > > > > > > https://github.com/apache/httpcomponents-client/blob/d8f702fb4d44c746bb0edf00643aa7139cb8bdf7/httpclient5/src/main/java/org/apache/hc/client5/http/ssl/SSLConnectionSocketFactory.java#L274 > > > > > > > > > > > > > > > > Now it's `false`: > > > > > > > > > > > > > > > > > https://github.com/apache/httpcomponents-client/blob/ffe4a05f4faf3706a41ff660bcb3474c6b5101a3/httpclient5/src/main/java/org/apache/hc/client5/http/ssl/AbstractClientTlsStrategy.java#L208 > > > > > > > > > > > > > > > > This was deliberately changed in this commit: > > > > > > > > > > > > > > > > > https://github.com/apache/httpcomponents-client/commit/ee0a102104d03a8d1cfe18e571d179c41242182c > > > > > > > > > > > > > > > > > > > > > > This change was introduced in response to HTTPCLIENT-2328 > > > > > > > [1]. > > > > > > > I will > > > > > > > review the code for accidental mistakes. Conceptually the > > > > > > > `autoClose` > > > > > > > setting should be merely a convenience. > > > > > > > > > > > > > > Oleg > > > > > > > > > > > > > > > > > > > > > [1] https://issues.apache.org/jira/browse/HTTPCLIENT-2328 > > > > > > > > > > > > > > > > > > > I think this change in core is more relevant [1]. At the > > > > > > moment I > > > > > > cannot see anything obviously wrong with it. > > > > > > > > > > > > Would you be able to share (contribute) a reproducer for the > > > > > > defect? > > > > > > > > > > > > Oleg > > > > > > > > > > > > [1] > > > > > > > https://github.com/apache/httpcomponents-core/commit/1c95b2994f22b91bc7f54cb66ed84a8e94111a61 > > > > > > > > > > > > > > > > > > > > > > > > > > > > On Thu, May 29, 2025 at 6:08 PM Ryan Schmitt > > > > > > > > <rschm...@apache.org> > > > > > > > > wrote: > > > > > > > > > > > > > > > > > > I've been debugging a load test regression, which > > > > > > > > > turned > > > > > > > > > out to > > > > > > > > > be > > > > > > > > > caused by a client instance leak. The test started > > > > > > > > > failing > > > > > > > > > on > > > > > > > > > 5.4.4 > > > > > > > > > due to running out of file descriptors. What I realized > > > > > > > > > is > > > > > > > > > that > > > > > > > > > leaked > > > > > > > > > connection pools are eventually cleaned up by garbage > > > > > > > > > collection, > > > > > > > > > which causes the sockets' file descriptors to be > > > > > > > > > released. > > > > > > > > > What > > > > > > > > > has > > > > > > > > > changed in 5.4.4 is that this process now takes TWO > > > > > > > > > rounds > > > > > > > > > of > > > > > > > > > garbage > > > > > > > > > collection. This can be seen in the `lsof` output, > > > > > > > > > which > > > > > > > > > shows > > > > > > > > > where > > > > > > > > > in the TCP state machine the socket is: > > > > > > > > > > > > > > > > > > 5.2: ESTABLISHED -(gc)-> (gone) > > > > > > > > > 5.4: ESTABLISHED -(gc)-> CLOSE_WAIT -(gc)-> (gone) > > > > > > > > > > > > > > > > > > Does anyone know what might have changed that would > > > > > > > > > cause > > > > > > > > > this? > > > > > > > > > I'm > > > > > > > > > specifically asking about the synchronous client (I > > > > > > > > > haven't > > > > > > > > > tested > > > > > > > > > the > > > > > > > > > async client). > > > > > > > > > > > > > > > > --------------------------------------------------------- > > > > > > > > ---- > > > > > > > > ------ > > > > > > > > -- > > > > > > > > To unsubscribe, e-mail: dev-unsubscr...@hc.apache.org > > > > > > > > For additional commands, e-mail: dev-h...@hc.apache.org > > > > > > > > > > > > > > > > > > > > > > > > > > > > > ----------------------------------------------------------- > > > > > > > ---- > > > > > > > ------ > > > > > > > To unsubscribe, e-mail: dev-unsubscr...@hc.apache.org > > > > > > > For additional commands, e-mail: dev-h...@hc.apache.org > > > > > > > > > > > > > > > > > > > > > > > > > ------------------------------------------------------------- > > > > > > ---- > > > > > > ---- > > > > > > To unsubscribe, e-mail: dev-unsubscr...@hc.apache.org > > > > > > For additional commands, e-mail: dev-h...@hc.apache.org > > > > > > > > > > > > > > ----------------------------------------------------------------- > > > > ---- > > > > To unsubscribe, e-mail: dev-unsubscr...@hc.apache.org > > > > For additional commands, e-mail: dev-h...@hc.apache.org > > > > > > > > > > > > > ------------------------------------------------------------------- > > > -- > > > To unsubscribe, e-mail: dev-unsubscr...@hc.apache.org > > > For additional commands, e-mail: dev-h...@hc.apache.org > > > > > > > --------------------------------------------------------------------- > > To unsubscribe, e-mail: dev-unsubscr...@hc.apache.org > > For additional commands, e-mail: dev-h...@hc.apache.org > > > > > --------------------------------------------------------------------- > To unsubscribe, e-mail: dev-unsubscr...@hc.apache.org > For additional commands, e-mail: dev-h...@hc.apache.org > >
