Thanks Mark. *What is the typical response size for one of these requests? * It' basically a dummy response JSON of ~300bytes. I expect 2300bytes of response in my production use case, but the purpose here was to isolate as many things as possible. Hence a dummy response.
*How long does a typical test take to process? * I see Tomcat's memory reaching 28G in about 2hours at 19K TPS. The number of streams on my client was 500 - which means about 40 connections per second. * What are the GC roots for those RequestInfo objects?* https://ibb.co/fMRmCXZ I hope I was able to answer everything as expected. Thanks. On Thu, Jun 25, 2020 at 8:30 PM Mark Thomas <ma...@apache.org> wrote: > Thanks. > > I've looked at the code and I have tried various tests but I am unable > to re-create a memory leak. > > The code used to (before I made a few changes this afternoon) retain a > lot more memory per Stream and it is possible that what you are seeing > is a system that doesn't have enough memory to achieve steady state. > > If you are able to build the latest 9.0.x and test that, that could be > helpful. Alternatively, I could provide a test build for you to > experiment with. > > Some additional questions that might aid understanding: > > - What is the typical response size for one of these requests? > - How long does a typical test take to process? > - What are the GC roots for those RequestInfo objects? > > Thanks again, > > Mark > > > > > On 25/06/2020 15:10, Chirag Dewan wrote: > > Hi Mark, > > > > Its the default APR connector with 150 Threads. > > > > Chirag > > > > On Thu, 25 Jun, 2020, 7:30 pm Mark Thomas, <ma...@apache.org> wrote: > > > >> On 25/06/2020 11:00, Chirag Dewan wrote: > >>> Thanks for the quick check Mark. > >>> > >>> These are the images I tried referring to: > >>> > >>> https://ibb.co/LzKtRgh > >>> > >>> https://ibb.co/2s7hqRL > >>> > >>> https://ibb.co/KmKj590 > >>> > >>> > >>> The last one is the MAT screenshot showing many RequestInfo objects. > >> > >> Thanks. That certainly looks like a memory leak. I'll take a closer > >> look. Out of interest, how many threads is the Connector configured to > use? > >> > >> Mark > >> > >> > >>> > >>> > >>> Thanks, > >>> > >>> Chirag > >>> > >>> On Wed, Jun 24, 2020 at 8:30 PM Mark Thomas <ma...@apache.org> wrote: > >>> > >>>> On 24/06/2020 12:17, Mark Thomas wrote: > >>>>> On 22/06/2020 11:06, Chirag Dewan wrote: > >>>>>> Hi, > >>>>>> > >>>>>> Update: We found that Tomcat goes OOM when a client closes and opens > >> new > >>>>>> connections every second. In the memory dump, we see a lot of > >>>>>> RequestInfo objects that are causing the memory spike. > >>>>>> > >>>>>> After a while, Tomcat goes OOM and start rejecting request(I get a > >>>>>> request timed out on my client). This seems like a bug to me. > >>>>>> > >>>>>> For better understanding, let me explain my use case again: > >>>>>> > >>>>>> I have a jetty client that sends HTTP2 requests to Tomcat. My > >>>>>> requirement is to close a connection after a configurable(say 5000) > >>>>>> number of requests/streams and open a new connection that continues > to > >>>>>> send requests. I close a connection by sending a GoAway frame. > >>>>>> > >>>>>> When I execute this use case under load, I see that after ~2hours my > >>>>>> requests fail and I get a series of errors like request > >>>>>> timeouts(5seconds), invalid window update frame, and connection > close > >>>>>> exception on my client. > >>>>>> On further debugging, I found that it's a Tomcat memory problem and > it > >>>>>> goes OOM after sometime under heavy load with multiple connections > >> being > >>>>>> re-established by the clients. > >>>>>> > >>>>>> image.png > >>>>>> > >>>>>> image.png > >>>>>> > >>>>>> Is this a known issue? Or a known behavior with Tomcat? > >>>>> > >>>>> Embedded images get dropped by the list software. Post those images > >>>>> somewhere we can see them. > >>>>> > >>>>>> Please let me know if you any experience with such a situation. > Thanks > >>>>>> in advance. > >>>>> > >>>>> Nothing comes to mind. > >>>>> > >>>>> I'll try some simple tests with HTTP/2. > >>>> > >>>> I don't see a memory leak (the memory is reclaimed eventually) but I > do > >>>> see possibilities to release memory associated with request processing > >>>> sooner. > >>>> > >>>> Right now you need to allocate more memory to the Java process to > enable > >>>> Tomcat to handle the HTTP/2 load it is presented with. > >>>> > >>>> It looks like a reasonable chunk of memory is released when the > >>>> Connection closes that could be released earlier when the associated > >>>> Stream closes. I'll take a look at what can be done in that area. In > the > >>>> meantime, reducing the number of Streams you allow on a Connection > >>>> before it is closed should reduce overall memory usage. > >>>> > >>>> Mark > >>>> > >>>> --------------------------------------------------------------------- > >>>> To unsubscribe, e-mail: users-unsubscr...@tomcat.apache.org > >>>> For additional commands, e-mail: users-h...@tomcat.apache.org > >>>> > >>>> > >>> > >> > >> > >> --------------------------------------------------------------------- > >> To unsubscribe, e-mail: users-unsubscr...@tomcat.apache.org > >> For additional commands, e-mail: users-h...@tomcat.apache.org > >> > >> > > > > > --------------------------------------------------------------------- > To unsubscribe, e-mail: users-unsubscr...@tomcat.apache.org > For additional commands, e-mail: users-h...@tomcat.apache.org > >
