Thanks for the update, Mark! We will keep using the discardRequestsAndResponses workaround for now.
Arjan ________________________________ From: Mark Thomas <ma...@apache.org> Sent: Friday, January 10, 2025 11:18 To: users@tomcat.apache.org <users@tomcat.apache.org> Subject: Re: Excessive memory usage for HTTP/2 requests On 09/01/2025 14:19, Mark Thomas wrote: > On 03/01/2025 07:44, Mark Thomas wrote: >> Arjan, >> >> This is the right place to ask that question. Taking a look at this is >> on my TODO list. Between addressing CVE-2024-50379 and CVE-2024-56337 >> and the holiday season I haven't got to it yet. I expect to look at it >> before the next release (it isn't the only issue on my TODO list). > > I've made some progress. I understand what is going on and I have a fix > that is working well in testing so far. I have some more testing to do. > If that goes well I expect to commit a fix later today for the next > round of releases. Just a quick follow-up on this. One of the changes is quite invasive so we are going to roll it out slowly just to be safe. The fix will be in the next 11.0.x release and then back-ported to 10.1.x and 9.0.x at least one release later. Using discardRequestsAndResponses continues to be a viable workaround. Note that with this fix in place, each request/response pair on the stack will still require ~60k. Mark > > Mark > >> >> Mark >> >> On 02/01/2025 14:06, Arjan van IJzendoorn wrote: >>> Hello, >>> >>> Am I asking questions in the wrong place or in the wrong way? I was >>> hoping to hear from someone here. Please point me in the right >>> direction if this is not the place. >>> >>> This "memory leak" may happen to others, too. >>> >>> Thanks, >>> Arjan >>> >>> >>> ________________________________ >>> From: Arjan van IJzendoorn <arjan.van.ijzendo...@blockbax.com.INVALID> >>> Sent: Thursday, December 19, 2024 15:16 >>> To: users@tomcat.apache.org <users@tomcat.apache.org> >>> Subject: Re: Excessive memory usage for HTTP/2 requests >>> >>> Hello, >>> >>> In my initial message, I tried embedding an image but that did not >>> work. Here is a link to a publicly hosted image instead: >>> >>> https://ibb.co/4KFXgdj >>> >>> In the memory analyzer screenshot, you can see the >>> 'recycledRequestAndResponses' stack taking up 288MB, which is more >>> than half of the total memory usage. >>> >>> Since the previous message, our service has been running >>> 'discardRequestsAndResponses' set to true and memory usage has been >>> stable. Is that option there to stay? The comment in the code of >>> Http2Protocol suggests that using the stack improves performance so I >>> am a little afraid that the option to turn this optimization off may >>> disappear in the future. >>> >>> We are still not sure why Request objects are so large in our case. >>> >>> Kind regards, Arjan >>> >>> >>> ________________________________ >>> From: Arjan van IJzendoorn <arjan.van.ijzendo...@blockbax.com> >>> Sent: Tuesday, December 17, 2024 13:41 >>> To: users@tomcat.apache.org <users@tomcat.apache.org> >>> Subject: Excessive memory usage for HTTP/2 requests >>> >>> Hello Tomcat friends, >>> >>> Our Spring Boot 3.4.0 application uses Tomcat 10.1.33. Recently we >>> started seeing the memory usage of our app grow and after three days >>> it would reach its memory limit and crash. >>> >>> We investigated heap dumps and found that a single data structure >>> inside Tomcat grew by 240MB in a single day. It is the >>> 'recycledRequestsAndResponses' stack inside Http2Protocol.java. This >>> data structure was introduced in the 3 latest commits on >>> Http2Protocol.java: >>> >>> https://github.com/apache/tomcat/ >>> commits/9669f73517971e2b45280979a63b8153585cddc8/java/org/apache/ >>> coyote/http2/Http2Protocol.java >>> >>> It was introduced in Tomcat 10.1.27 (not released) and released as >>> part of Tomcat 10.1.28: >>> >>>> Align HTTP/2 with HTTP/1.1 and recycle the container internal >>>> request and response processing objects by default. This behavior >>>> can be controlled via the new discardRequestsAndResponses attribute >>>> on the HTTP/2 upgrade protocol. (markt) >>> It only applies to HTTP/2 requests, and we get a steady stream of >>> those from other internal services. Requests from the outside world >>> are always HTTP 1.1. As you can see in the Memory Analyzer >>> screenshot, each request in the stack is rather large (360KB). There >>> are byte arrays of 65535 bytes that are only filled with 200 characters. >>> >>> There is a limit to the stack which is set in this line: >>> >>>> recycledRequestsAndResponses.setLimit(http11Protocol.getMaxConnections()); >>> In our case the maximum number of connections is 8192. So. the stack >>> could theoretically become as large as 8192 x 360KB (almost 3 Gigabyte). >>> >>> I do see code to "recyle" (i.e. clear) the request before putting it >>> onto the stack: >>> >>>> requestAndResponse.recycle(); >>> This clears a lot of fields but (probably intentionally) not some of >>> the larger fields. >>> >>> Fortunately, we can disable this mechanism, and we have just deployed >>> a version of our app with 'discardRequestsAndResponses' set to true >>> by customizing the 'TomcatServletWebServerFactory'. After running >>> for two and a half hours, we can already see that our app now >>> consumes memory at a slower rate. >>> >>> Are we doing something wrong? Is the maximum number of connections >>> unreasonably high? Is there some reason why our Request objects >>> become so large? >>> >>> Kind regards, Arjan van IJzendoorn >>> >>> >>> [cid:21200d4c-68ac-46b8-8dc8-affc43f6475d] >>> >>> >>> >> >> --------------------------------------------------------------------- >> To unsubscribe, e-mail: users-unsubscr...@tomcat.apache.org >> For additional commands, e-mail: users-h...@tomcat.apache.org >> > > > --------------------------------------------------------------------- > To unsubscribe, e-mail: users-unsubscr...@tomcat.apache.org > For additional commands, e-mail: users-h...@tomcat.apache.org > --------------------------------------------------------------------- To unsubscribe, e-mail: users-unsubscr...@tomcat.apache.org For additional commands, e-mail: users-h...@tomcat.apache.org
