Thanks. I've looked at the code and I have tried various tests but I am unable to re-create a memory leak.
The code used to (before I made a few changes this afternoon) retain a lot more memory per Stream and it is possible that what you are seeing is a system that doesn't have enough memory to achieve steady state. If you are able to build the latest 9.0.x and test that, that could be helpful. Alternatively, I could provide a test build for you to experiment with. Some additional questions that might aid understanding: - What is the typical response size for one of these requests? - How long does a typical test take to process? - What are the GC roots for those RequestInfo objects? Thanks again, Mark On 25/06/2020 15:10, Chirag Dewan wrote: > Hi Mark, > > Its the default APR connector with 150 Threads. > > Chirag > > On Thu, 25 Jun, 2020, 7:30 pm Mark Thomas, <ma...@apache.org> wrote: > >> On 25/06/2020 11:00, Chirag Dewan wrote: >>> Thanks for the quick check Mark. >>> >>> These are the images I tried referring to: >>> >>> https://ibb.co/LzKtRgh >>> >>> https://ibb.co/2s7hqRL >>> >>> https://ibb.co/KmKj590 >>> >>> >>> The last one is the MAT screenshot showing many RequestInfo objects. >> >> Thanks. That certainly looks like a memory leak. I'll take a closer >> look. Out of interest, how many threads is the Connector configured to use? >> >> Mark >> >> >>> >>> >>> Thanks, >>> >>> Chirag >>> >>> On Wed, Jun 24, 2020 at 8:30 PM Mark Thomas <ma...@apache.org> wrote: >>> >>>> On 24/06/2020 12:17, Mark Thomas wrote: >>>>> On 22/06/2020 11:06, Chirag Dewan wrote: >>>>>> Hi, >>>>>> >>>>>> Update: We found that Tomcat goes OOM when a client closes and opens >> new >>>>>> connections every second. In the memory dump, we see a lot of >>>>>> RequestInfo objects that are causing the memory spike. >>>>>> >>>>>> After a while, Tomcat goes OOM and start rejecting request(I get a >>>>>> request timed out on my client). This seems like a bug to me. >>>>>> >>>>>> For better understanding, let me explain my use case again: >>>>>> >>>>>> I have a jetty client that sends HTTP2 requests to Tomcat. My >>>>>> requirement is to close a connection after a configurable(say 5000) >>>>>> number of requests/streams and open a new connection that continues to >>>>>> send requests. I close a connection by sending a GoAway frame. >>>>>> >>>>>> When I execute this use case under load, I see that after ~2hours my >>>>>> requests fail and I get a series of errors like request >>>>>> timeouts(5seconds), invalid window update frame, and connection close >>>>>> exception on my client. >>>>>> On further debugging, I found that it's a Tomcat memory problem and it >>>>>> goes OOM after sometime under heavy load with multiple connections >> being >>>>>> re-established by the clients. >>>>>> >>>>>> image.png >>>>>> >>>>>> image.png >>>>>> >>>>>> Is this a known issue? Or a known behavior with Tomcat? >>>>> >>>>> Embedded images get dropped by the list software. Post those images >>>>> somewhere we can see them. >>>>> >>>>>> Please let me know if you any experience with such a situation. Thanks >>>>>> in advance. >>>>> >>>>> Nothing comes to mind. >>>>> >>>>> I'll try some simple tests with HTTP/2. >>>> >>>> I don't see a memory leak (the memory is reclaimed eventually) but I do >>>> see possibilities to release memory associated with request processing >>>> sooner. >>>> >>>> Right now you need to allocate more memory to the Java process to enable >>>> Tomcat to handle the HTTP/2 load it is presented with. >>>> >>>> It looks like a reasonable chunk of memory is released when the >>>> Connection closes that could be released earlier when the associated >>>> Stream closes. I'll take a look at what can be done in that area. In the >>>> meantime, reducing the number of Streams you allow on a Connection >>>> before it is closed should reduce overall memory usage. >>>> >>>> Mark >>>> >>>> --------------------------------------------------------------------- >>>> To unsubscribe, e-mail: users-unsubscr...@tomcat.apache.org >>>> For additional commands, e-mail: users-h...@tomcat.apache.org >>>> >>>> >>> >> >> >> --------------------------------------------------------------------- >> To unsubscribe, e-mail: users-unsubscr...@tomcat.apache.org >> For additional commands, e-mail: users-h...@tomcat.apache.org >> >> > --------------------------------------------------------------------- To unsubscribe, e-mail: users-unsubscr...@tomcat.apache.org For additional commands, e-mail: users-h...@tomcat.apache.org
