Hello, Am I asking questions in the wrong place or in the wrong way? I was hoping to hear from someone here. Please point me in the right direction if this is not the place.
This "memory leak" may happen to others, too. Thanks, Arjan ________________________________ From: Arjan van IJzendoorn <arjan.van.ijzendo...@blockbax.com.INVALID> Sent: Thursday, December 19, 2024 15:16 To: users@tomcat.apache.org <users@tomcat.apache.org> Subject: Re: Excessive memory usage for HTTP/2 requests Hello, In my initial message, I tried embedding an image but that did not work. Here is a link to a publicly hosted image instead: https://ibb.co/4KFXgdj In the memory analyzer screenshot, you can see the 'recycledRequestAndResponses' stack taking up 288MB, which is more than half of the total memory usage. Since the previous message, our service has been running 'discardRequestsAndResponses' set to true and memory usage has been stable. Is that option there to stay? The comment in the code of Http2Protocol suggests that using the stack improves performance so I am a little afraid that the option to turn this optimization off may disappear in the future. We are still not sure why Request objects are so large in our case. Kind regards, Arjan ________________________________ From: Arjan van IJzendoorn <arjan.van.ijzendo...@blockbax.com> Sent: Tuesday, December 17, 2024 13:41 To: users@tomcat.apache.org <users@tomcat.apache.org> Subject: Excessive memory usage for HTTP/2 requests Hello Tomcat friends, Our Spring Boot 3.4.0 application uses Tomcat 10.1.33. Recently we started seeing the memory usage of our app grow and after three days it would reach its memory limit and crash. We investigated heap dumps and found that a single data structure inside Tomcat grew by 240MB in a single day. It is the 'recycledRequestsAndResponses' stack inside Http2Protocol.java. This data structure was introduced in the 3 latest commits on Http2Protocol.java: https://github.com/apache/tomcat/commits/9669f73517971e2b45280979a63b8153585cddc8/java/org/apache/coyote/http2/Http2Protocol.java It was introduced in Tomcat 10.1.27 (not released) and released as part of Tomcat 10.1.28: > Align HTTP/2 with HTTP/1.1 and recycle the container internal request and > response processing objects by default. This behavior can be controlled via > the new discardRequestsAndResponses attribute on the HTTP/2 upgrade protocol. > (markt) It only applies to HTTP/2 requests, and we get a steady stream of those from other internal services. Requests from the outside world are always HTTP 1.1. As you can see in the Memory Analyzer screenshot, each request in the stack is rather large (360KB). There are byte arrays of 65535 bytes that are only filled with 200 characters. There is a limit to the stack which is set in this line: > recycledRequestsAndResponses.setLimit(http11Protocol.getMaxConnections()); In our case the maximum number of connections is 8192. So. the stack could theoretically become as large as 8192 x 360KB (almost 3 Gigabyte). I do see code to "recyle" (i.e. clear) the request before putting it onto the stack: > requestAndResponse.recycle(); This clears a lot of fields but (probably intentionally) not some of the larger fields. Fortunately, we can disable this mechanism, and we have just deployed a version of our app with 'discardRequestsAndResponses' set to true by customizing the 'TomcatServletWebServerFactory'. After running for two and a half hours, we can already see that our app now consumes memory at a slower rate. Are we doing something wrong? Is the maximum number of connections unreasonably high? Is there some reason why our Request objects become so large? Kind regards, Arjan van IJzendoorn [cid:21200d4c-68ac-46b8-8dc8-affc43f6475d]
