Hi, On Wed, Sep 30, 2020 at 11:35 AM Mark Thomas <ma...@apache.org> wrote:
> On 30/09/2020 06:42, Arshiya Shariff wrote: > > Hi Martin , > > > > Thank you for the response. > > > > With a payload of 200 bytes we were able to send 20K requests/sec with > 200 users from Jmeter without any memory issue . On increasing the payload > to 5Kb and the number of users to 1000 in Jmeter and sending 1000 requests > per second , the heap of 20GB got filled in 2 minutes . With 200 users the > memory is cleared in the G1 mixed GC itself , but with 1000 users the > memory is not cleared in the mixed GC , it takes full GCs of 7 to 10 > seconds to clear the memory. These cases were executed with maxThreads 200 > in tomcat , so we tried increasing the maxThreads from 200 to 1000, but > still GC was struggling . > > > > When we tried with 10 instances of JMeter , each with 100 users , where > each instance was started with a delay of 1 minute we were able to see 1000 > connections created in tomcat without any memory issues. But when 1000 > users are created using single instance of JMeter in 20 seconds , tomcat's > memory is filling fast- 20GB in 2 minutes. > > We suspect that the burst of connections being opened has a problem . > Please help us with the same . > > > > On analyzing the heap dump we see > org.apache.tomcat.util.collections.SynchronizedStack occupying around 93% > of 3GB live data ,the remaining 17GB is Garbage collected in the heap dump. > > You can't have high throughput, low GC pauses and small heap sizes. > Broadly you can have any two of those three at the expense of the third. > > The way Tomcat currently retains information about completed h2 streams > means you are likely to need a large heap under heavy load. There are > some changes already in 10.0.x that I plan to back-port to 9.0.x and > 8.5.x later today that should significantly reduce the heap requirements. > Here is a screenshot of me loading Tomcat HTTP2 9.0.x+the changes from 10.0.x with Vegeta for 3 mins: https://pasteboard.co/JtshrAs.png As you can see the GC is properly cleaning the heap. At the end the memory is not released until the GC kicks. Note: this is with a GET request without a body! I'm going to start a new email thread for POST with body - there I get GOAWAY+ENHANCE_YOUR_CALM for very low load. Martin > > Mark > > > > > > Thanks and Regards > > Arshiya Shariff > > > > -----Original Message----- > > From: Martin Grigorov <mgrigo...@apache.org> > > Sent: Monday, September 28, 2020 11:44 PM > > To: Tomcat Users List <users@tomcat.apache.org> > > Subject: Re: HTTP2: memory filled up fast on increasing the connections > to 1000/2000 (Embedded tomcat 9.0.38) > > > > Hi Arshiya, > > > > > > On Mon, Sep 28, 2020 at 7:59 PM Arshiya Shariff < > arshiya.shar...@ericsson.com.invalid> wrote: > > > >> Hi All, > >> With 200 threads(users) , ramp up duration of 2 seconds , loop count > >> 80 and by sending 1000 http2 requests/sec from JMeter Client to an > >> embedded tomcat application we did not observe any memory issue , but > >> on sending > >> 1000 http2 requests/sec with 2000 or 1000 users from JMeter , the > >> application's heap space of 20 GB is occupied in 2 minutes and after 2 > >> full GCs the memory clears and comes down to 4GB (expected) . > >> > > > > I am not sure whether you follow the other discussions at users@. > > In another email thread we discuss load testing Tomcat HTTP2 and we are > able to make around 12K reqs/s with another load testing tool - > https://protect2.fireeye.com/v1/url?k=f8cfc13c-a66f0379-f8cf81a7-8692dc8284cb-2c0aae53194b790f&q=1&e=6a9c569d-7da1-4394-a9ac-bf72724992fa&u=https%3A%2F%2Fgithub.com%2Ftsenart%2Fvegeta > > For me JMeter itself failed with OOM when increasing the number of the > virtual users above 2K. > > There are several improvements in Tomcat master and 9.0.x in the HTTP2 > area. Some of the changes are not yet downported to 9.0.x. We still test > them, trying to avoid introducing regressions in 9.0.x. > > > > > >> > >> Embedded tomcat Version:9.0.38 > >> Max Threads : 200 > >> > > > > The number of threads should be less if you do only CPU calculations > without IO/network. If your app blocks on IO/network calls then you need > more spare threads. > > With more threads there will be more context switches and less > throughput. > > That's why there is no one golden rule that applies to all applications. > > 200 is a good default that works for most of the applications. But you > need to test with different values to see which one gives the best > performance for your scenaria. > > > > > >> All other properties are the tomcat defaults. > >> > >> Why is tomcat not able to process many connections ? > >> > > > > You can tell us by enabling -XX:+HeapDumpOnOutOfMemoryError and > -XX:HeapDumpPath=<file-or-dir-path>. Once you have the .hprof file you can > examine it with Eclipse Memory Analyzer tool and see what is leaking. > > I will try to reproduce this issue tomorrow with Vegeta. > > > > > >> Why is the memory filled when the connections are increased, are there > >> any parameters to tune connections ? > >> Please let us know. > >> > >> Thanks and Regards > >> Arshiya Shariff > >> > > > > --------------------------------------------------------------------- > > To unsubscribe, e-mail: users-unsubscr...@tomcat.apache.org > > For additional commands, e-mail: users-h...@tomcat.apache.org > > > > > --------------------------------------------------------------------- > To unsubscribe, e-mail: users-unsubscr...@tomcat.apache.org > For additional commands, e-mail: users-h...@tomcat.apache.org > >
