On Wed, Sep 30, 2020 at 12:50 PM Martin Grigorov <mgrigo...@apache.org> wrote:
> Hi, > > On Wed, Sep 30, 2020 at 11:35 AM Mark Thomas <ma...@apache.org> wrote: > >> On 30/09/2020 06:42, Arshiya Shariff wrote: >> > Hi Martin , >> > >> > Thank you for the response. >> > >> > With a payload of 200 bytes we were able to send 20K requests/sec with >> 200 users from Jmeter without any memory issue . On increasing the payload >> to 5Kb and the number of users to 1000 in Jmeter and sending 1000 requests >> per second , the heap of 20GB got filled in 2 minutes . With 200 users the >> memory is cleared in the G1 mixed GC itself , but with 1000 users the >> memory is not cleared in the mixed GC , it takes full GCs of 7 to 10 >> seconds to clear the memory. These cases were executed with maxThreads 200 >> in tomcat , so we tried increasing the maxThreads from 200 to 1000, but >> still GC was struggling . >> > >> > When we tried with 10 instances of JMeter , each with 100 users , where >> each instance was started with a delay of 1 minute we were able to see 1000 >> connections created in tomcat without any memory issues. But when 1000 >> users are created using single instance of JMeter in 20 seconds , tomcat's >> memory is filling fast- 20GB in 2 minutes. >> > We suspect that the burst of connections being opened has a problem . >> Please help us with the same . >> > >> > On analyzing the heap dump we see >> org.apache.tomcat.util.collections.SynchronizedStack occupying around 93% >> of 3GB live data ,the remaining 17GB is Garbage collected in the heap dump. >> >> You can't have high throughput, low GC pauses and small heap sizes. >> Broadly you can have any two of those three at the expense of the third. >> >> The way Tomcat currently retains information about completed h2 streams >> means you are likely to need a large heap under heavy load. There are >> some changes already in 10.0.x that I plan to back-port to 9.0.x and >> 8.5.x later today that should significantly reduce the heap requirements. >> > > Here is a screenshot of me loading Tomcat HTTP2 9.0.x+the changes from > 10.0.x with Vegeta for 3 mins: > https://pasteboard.co/JtshrAs.png > As you can see the GC is properly cleaning the heap. At the end the memory > is not released until the GC kicks. > > Note: this is with a GET request without a body! I'm going to start a new > email thread for POST with body - there I get GOAWAY+ENHANCE_YOUR_CALM for > very low load. > When I load test HTTP2 with POST (with big bodies) there are many errors like: 1) Exception in thread "https-jsse-nio-8080-exec-5" java.nio.BufferOverflowException at java.base/java.nio.ByteBuffer.put(ByteBuffer.java:957) at java.base/java.nio.HeapByteBuffer.put(HeapByteBuffer.java:247) at org.apache.tomcat.util.net.SocketBufferHandler.unReadReadBuffer(SocketBufferHandler.java:100) at org.apache.tomcat.util.net.SocketWrapperBase.unRead(SocketWrapperBase.java:401) at org.apache.coyote.http2.Http2AsyncParser$FrameCompletionHandler.completed(Http2AsyncParser.java:307) at org.apache.coyote.http2.Http2AsyncParser$FrameCompletionHandler.completed(Http2AsyncParser.java:164) at org.apache.tomcat.util.net.SocketWrapperBase$VectoredIOCompletionHandler.completed(SocketWrapperBase.java:1087) at org.apache.tomcat.util.net.NioEndpoint$NioSocketWrapper$NioOperationState.run(NioEndpoint.java:1511) at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1130) at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:630) at org.apache.tomcat.util.threads.TaskThread$WrappingRunnable.run(TaskThread.java:61) at java.base/java.lang.Thread.run(Thread.java:832) 2) Sep 30, 2020 3:44:04 PM org.apache.tomcat.util.net.NioEndpoint$Poller events SEVERE: Failed to register socket with selector from poller java.nio.channels.ClosedChannelException at java.base/java.nio.channels.spi.AbstractSelectableChannel.register(AbstractSelectableChannel.java:222) at org.apache.tomcat.util.net.NioEndpoint$Poller.events(NioEndpoint.java:609) at org.apache.tomcat.util.net.NioEndpoint$Poller.run(NioEndpoint.java:703) at java.base/java.lang.Thread.run(Thread.java:832) And the client gets these kind of error responses: "Post \"https://localhost:8080/testbed/plaintext\": http2: server sent GOAWAY and closed the connection; LastStreamID=7, ErrCode=PROTOCOL_ERROR, debug=\"Connection [9354], Stream [7], The content length header value [255] does not agree with the size of the data received [0]\"", "Post \"https://localhost:8080/testbed/plaintext\": http2: server sent GOAWAY and closed the connection; LastStreamID=31, ErrCode=ENHANCE_YOUR_CALM, debug=\"Connection [9355], Too much overhead so the connection will be closed\"" "Post \"https://localhost:8080/testbed/plaintext\": http2: server sent GOAWAY and closed the connection; LastStreamID=6155, ErrCode=STREAM_CLOSED, debug=\"Connection [10048], Stream [6153], State [CLOSED_RX], Frame type [RST]\"" "status_codes": { "0": 286, "200": 35373 } , i.e. just a small portion of the requests fail. If I tell Vegeta to use HTTP1.1 (-http2=f) then everything is OK. It could be a problem in Vegeta too. The memory usage is just fine (as in the chart for GET). Martin > Martin > > >> >> Mark >> >> >> > >> > Thanks and Regards >> > Arshiya Shariff >> > >> > -----Original Message----- >> > From: Martin Grigorov <mgrigo...@apache.org> >> > Sent: Monday, September 28, 2020 11:44 PM >> > To: Tomcat Users List <users@tomcat.apache.org> >> > Subject: Re: HTTP2: memory filled up fast on increasing the connections >> to 1000/2000 (Embedded tomcat 9.0.38) >> > >> > Hi Arshiya, >> > >> > >> > On Mon, Sep 28, 2020 at 7:59 PM Arshiya Shariff < >> arshiya.shar...@ericsson.com.invalid> wrote: >> > >> >> Hi All, >> >> With 200 threads(users) , ramp up duration of 2 seconds , loop count >> >> 80 and by sending 1000 http2 requests/sec from JMeter Client to an >> >> embedded tomcat application we did not observe any memory issue , but >> >> on sending >> >> 1000 http2 requests/sec with 2000 or 1000 users from JMeter , the >> >> application's heap space of 20 GB is occupied in 2 minutes and after 2 >> >> full GCs the memory clears and comes down to 4GB (expected) . >> >> >> > >> > I am not sure whether you follow the other discussions at users@. >> > In another email thread we discuss load testing Tomcat HTTP2 and we are >> able to make around 12K reqs/s with another load testing tool - >> https://protect2.fireeye.com/v1/url?k=f8cfc13c-a66f0379-f8cf81a7-8692dc8284cb-2c0aae53194b790f&q=1&e=6a9c569d-7da1-4394-a9ac-bf72724992fa&u=https%3A%2F%2Fgithub.com%2Ftsenart%2Fvegeta >> > For me JMeter itself failed with OOM when increasing the number of the >> virtual users above 2K. >> > There are several improvements in Tomcat master and 9.0.x in the HTTP2 >> area. Some of the changes are not yet downported to 9.0.x. We still test >> them, trying to avoid introducing regressions in 9.0.x. >> > >> > >> >> >> >> Embedded tomcat Version:9.0.38 >> >> Max Threads : 200 >> >> >> > >> > The number of threads should be less if you do only CPU calculations >> without IO/network. If your app blocks on IO/network calls then you need >> more spare threads. >> > With more threads there will be more context switches and less >> throughput. >> > That's why there is no one golden rule that applies to all applications. >> > 200 is a good default that works for most of the applications. But you >> need to test with different values to see which one gives the best >> performance for your scenaria. >> > >> > >> >> All other properties are the tomcat defaults. >> >> >> >> Why is tomcat not able to process many connections ? >> >> >> > >> > You can tell us by enabling -XX:+HeapDumpOnOutOfMemoryError and >> -XX:HeapDumpPath=<file-or-dir-path>. Once you have the .hprof file you can >> examine it with Eclipse Memory Analyzer tool and see what is leaking. >> > I will try to reproduce this issue tomorrow with Vegeta. >> > >> > >> >> Why is the memory filled when the connections are increased, are there >> >> any parameters to tune connections ? >> >> Please let us know. >> >> >> >> Thanks and Regards >> >> Arshiya Shariff >> >> >> > >> > --------------------------------------------------------------------- >> > To unsubscribe, e-mail: users-unsubscr...@tomcat.apache.org >> > For additional commands, e-mail: users-h...@tomcat.apache.org >> > >> >> >> --------------------------------------------------------------------- >> To unsubscribe, e-mail: users-unsubscr...@tomcat.apache.org >> For additional commands, e-mail: users-h...@tomcat.apache.org >> >>
