On 30/09/2020 06:42, Arshiya Shariff wrote: > Hi Martin , > > Thank you for the response. > > With a payload of 200 bytes we were able to send 20K requests/sec with 200 > users from Jmeter without any memory issue . On increasing the payload to 5Kb > and the number of users to 1000 in Jmeter and sending 1000 requests per > second , the heap of 20GB got filled in 2 minutes . With 200 users the memory > is cleared in the G1 mixed GC itself , but with 1000 users the memory is not > cleared in the mixed GC , it takes full GCs of 7 to 10 seconds to clear the > memory. These cases were executed with maxThreads 200 in tomcat , so we tried > increasing the maxThreads from 200 to 1000, but still GC was struggling . > > When we tried with 10 instances of JMeter , each with 100 users , where each > instance was started with a delay of 1 minute we were able to see 1000 > connections created in tomcat without any memory issues. But when 1000 users > are created using single instance of JMeter in 20 seconds , tomcat's memory > is filling fast- 20GB in 2 minutes. > We suspect that the burst of connections being opened has a problem . Please > help us with the same . > > On analyzing the heap dump we see > org.apache.tomcat.util.collections.SynchronizedStack occupying around 93% of > 3GB live data ,the remaining 17GB is Garbage collected in the heap dump.
You can't have high throughput, low GC pauses and small heap sizes. Broadly you can have any two of those three at the expense of the third. The way Tomcat currently retains information about completed h2 streams means you are likely to need a large heap under heavy load. There are some changes already in 10.0.x that I plan to back-port to 9.0.x and 8.5.x later today that should significantly reduce the heap requirements. Mark > > Thanks and Regards > Arshiya Shariff > > -----Original Message----- > From: Martin Grigorov <mgrigo...@apache.org> > Sent: Monday, September 28, 2020 11:44 PM > To: Tomcat Users List <users@tomcat.apache.org> > Subject: Re: HTTP2: memory filled up fast on increasing the connections to > 1000/2000 (Embedded tomcat 9.0.38) > > Hi Arshiya, > > > On Mon, Sep 28, 2020 at 7:59 PM Arshiya Shariff > <arshiya.shar...@ericsson.com.invalid> wrote: > >> Hi All, >> With 200 threads(users) , ramp up duration of 2 seconds , loop count >> 80 and by sending 1000 http2 requests/sec from JMeter Client to an >> embedded tomcat application we did not observe any memory issue , but >> on sending >> 1000 http2 requests/sec with 2000 or 1000 users from JMeter , the >> application's heap space of 20 GB is occupied in 2 minutes and after 2 >> full GCs the memory clears and comes down to 4GB (expected) . >> > > I am not sure whether you follow the other discussions at users@. > In another email thread we discuss load testing Tomcat HTTP2 and we are able > to make around 12K reqs/s with another load testing tool - > https://protect2.fireeye.com/v1/url?k=f8cfc13c-a66f0379-f8cf81a7-8692dc8284cb-2c0aae53194b790f&q=1&e=6a9c569d-7da1-4394-a9ac-bf72724992fa&u=https%3A%2F%2Fgithub.com%2Ftsenart%2Fvegeta > For me JMeter itself failed with OOM when increasing the number of the > virtual users above 2K. > There are several improvements in Tomcat master and 9.0.x in the HTTP2 area. > Some of the changes are not yet downported to 9.0.x. We still test them, > trying to avoid introducing regressions in 9.0.x. > > >> >> Embedded tomcat Version:9.0.38 >> Max Threads : 200 >> > > The number of threads should be less if you do only CPU calculations without > IO/network. If your app blocks on IO/network calls then you need more spare > threads. > With more threads there will be more context switches and less throughput. > That's why there is no one golden rule that applies to all applications. > 200 is a good default that works for most of the applications. But you need > to test with different values to see which one gives the best performance for > your scenaria. > > >> All other properties are the tomcat defaults. >> >> Why is tomcat not able to process many connections ? >> > > You can tell us by enabling -XX:+HeapDumpOnOutOfMemoryError and > -XX:HeapDumpPath=<file-or-dir-path>. Once you have the .hprof file you can > examine it with Eclipse Memory Analyzer tool and see what is leaking. > I will try to reproduce this issue tomorrow with Vegeta. > > >> Why is the memory filled when the connections are increased, are there >> any parameters to tune connections ? >> Please let us know. >> >> Thanks and Regards >> Arshiya Shariff >> > > --------------------------------------------------------------------- > To unsubscribe, e-mail: users-unsubscr...@tomcat.apache.org > For additional commands, e-mail: users-h...@tomcat.apache.org > --------------------------------------------------------------------- To unsubscribe, e-mail: users-unsubscr...@tomcat.apache.org For additional commands, e-mail: users-h...@tomcat.apache.org
