On 03/01/2025 07:44, Mark Thomas wrote:
Arjan,
This is the right place to ask that question. Taking a look at this is
on my TODO list. Between addressing CVE-2024-50379 and CVE-2024-56337
and the holiday season I haven't got to it yet. I expect to look at it
before the next release (it isn't the only issue on my TODO list).
I've made some progress. I understand what is going on and I have a fix
that is working well in testing so far. I have some more testing to do.
If that goes well I expect to commit a fix later today for the next
round of releases.
Mark
Mark
On 02/01/2025 14:06, Arjan van IJzendoorn wrote:
Hello,
Am I asking questions in the wrong place or in the wrong way? I was
hoping to hear from someone here. Please point me in the right
direction if this is not the place.
This "memory leak" may happen to others, too.
Thanks,
Arjan
________________________________
From: Arjan van IJzendoorn <arjan.van.ijzendo...@blockbax.com.INVALID>
Sent: Thursday, December 19, 2024 15:16
To: users@tomcat.apache.org <users@tomcat.apache.org>
Subject: Re: Excessive memory usage for HTTP/2 requests
Hello,
In my initial message, I tried embedding an image but that did not
work. Here is a link to a publicly hosted image instead:
https://ibb.co/4KFXgdj
In the memory analyzer screenshot, you can see the
'recycledRequestAndResponses' stack taking up 288MB, which is more
than half of the total memory usage.
Since the previous message, our service has been running
'discardRequestsAndResponses' set to true and memory usage has been
stable. Is that option there to stay? The comment in the code of
Http2Protocol suggests that using the stack improves performance so I
am a little afraid that the option to turn this optimization off may
disappear in the future.
We are still not sure why Request objects are so large in our case.
Kind regards, Arjan
________________________________
From: Arjan van IJzendoorn <arjan.van.ijzendo...@blockbax.com>
Sent: Tuesday, December 17, 2024 13:41
To: users@tomcat.apache.org <users@tomcat.apache.org>
Subject: Excessive memory usage for HTTP/2 requests
Hello Tomcat friends,
Our Spring Boot 3.4.0 application uses Tomcat 10.1.33. Recently we
started seeing the memory usage of our app grow and after three days
it would reach its memory limit and crash.
We investigated heap dumps and found that a single data structure
inside Tomcat grew by 240MB in a single day. It is the
'recycledRequestsAndResponses' stack inside Http2Protocol.java. This
data structure was introduced in the 3 latest commits on
Http2Protocol.java:
https://github.com/apache/tomcat/
commits/9669f73517971e2b45280979a63b8153585cddc8/java/org/apache/
coyote/http2/Http2Protocol.java
It was introduced in Tomcat 10.1.27 (not released) and released as
part of Tomcat 10.1.28:
Align HTTP/2 with HTTP/1.1 and recycle the container internal request
and response processing objects by default. This behavior can be
controlled via the new discardRequestsAndResponses attribute on the
HTTP/2 upgrade protocol. (markt)
It only applies to HTTP/2 requests, and we get a steady stream of
those from other internal services. Requests from the outside world
are always HTTP 1.1. As you can see in the Memory Analyzer screenshot,
each request in the stack is rather large (360KB). There are byte
arrays of 65535 bytes that are only filled with 200 characters.
There is a limit to the stack which is set in this line:
recycledRequestsAndResponses.setLimit(http11Protocol.getMaxConnections());
In our case the maximum number of connections is 8192. So. the stack
could theoretically become as large as 8192 x 360KB (almost 3 Gigabyte).
I do see code to "recyle" (i.e. clear) the request before putting it
onto the stack:
requestAndResponse.recycle();
This clears a lot of fields but (probably intentionally) not some of
the larger fields.
Fortunately, we can disable this mechanism, and we have just deployed
a version of our app with 'discardRequestsAndResponses' set to true by
customizing the 'TomcatServletWebServerFactory'. After running for
two and a half hours, we can already see that our app now consumes
memory at a slower rate.
Are we doing something wrong? Is the maximum number of connections
unreasonably high? Is there some reason why our Request objects become
so large?
Kind regards, Arjan van IJzendoorn
[cid:21200d4c-68ac-46b8-8dc8-affc43f6475d]
---------------------------------------------------------------------
To unsubscribe, e-mail: users-unsubscr...@tomcat.apache.org
For additional commands, e-mail: users-h...@tomcat.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: users-unsubscr...@tomcat.apache.org
For additional commands, e-mail: users-h...@tomcat.apache.org
