Hi Mark , The issue is reproduced with version 9.0.39 as well. Max threads in Tomcat is 200.
Please find the case: Client:JMeter 5.2.1 (With http2 plugin) TPS: around 20 No of users from JMeter : 700 Message payload size: 6 KB to 34 KB Loop: Infinite We let the loop run infinitely and see the java.lang.StackOverflowError trace printed multiple times in the log within few minutes of starting the test. Please help us with this . What is the impact of StackOverflowError ? Thanks and Regards Arshiya Shariff -----Original Message----- From: Mark Thomas <ma...@apache.org> Sent: Friday, October 9, 2020 5:31 PM To: users@tomcat.apache.org Subject: Re: HTTP2: memory filled up fast on increasing the connections to 1000/2000 (Embedded tomcat 9.0.38) On 09/10/2020 12:32, Arshiya Shariff wrote: > Hi, > > Mark , with the test runs that I performed over clean 9.0.x branch I was not > able to reproduce this. Good. But I'd really like to understand why... > But with 9.0.38 and the jars built from 9.0.x with hash: > c8ec2d4cde3a31b0e9df9a30e7915d77ba725545 , with 700 or 1000 users > (connections) and on sending 1000 Requests per second (or even lesser) , > payload of 16K from JMeter I can see that this Exception occurs within few > minutes of starting the test . The maxThreads configured in tomcat is 200 . > > How often do you see these errors in your test run? > Randomly, at times 2 or 3 such traces. OK. Definitely a timing issue then. > Do you have the other end of that stack trace? > It is only the two lines that is recursively printed till the end about ~500 > times in one trace : > at > org.apache.tomcat.util.net.NioEndpoint$NioSocketWrapper$NioOperationState.run(NioEndpoint.java:1511) > at > org.apache.tomcat.util.net.SocketWrapperBase$VectoredIOCompletionHandl > er.completed(SocketWrapperBase.java:1100) Doesn't tell me much unfortunately. > I see the trace starting with : > Exception in thread "http-nio-x.y.z-1090-exec-107" > java.lang.StackOverflowError > at sun.nio.ch.SocketChannelImpl.read(SocketChannelImpl.java:446) > at org.apache.tomcat.util.net.NioChannel.read(NioChannel.java:174) > at > org.apache.tomcat.util.net.NioEndpoint$NioSocketWrapper$NioOperationState.run(NioEndpoint.java:1468) > at > org.apache.tomcat.util.net.SocketWrapperBase$VectoredIOCompletionHandl > er.completed(SocketWrapperBase.java:1100) > > (OR) > > Exception in thread "http-nio-x.y.z-1090-exec-87" java.lang.StackOverflowError > at sun.nio.ch.IOVecWrapper.get(IOVecWrapper.java:96) > at sun.nio.ch.IOUtil.read(IOUtil.java:240) > at sun.nio.ch.SocketChannelImpl.read(SocketChannelImpl.java:440) > at org.apache.tomcat.util.net.NioChannel.read(NioChannel.java:174) > at > org.apache.tomcat.util.net.NioEndpoint$NioSocketWrapper$NioOperationState.run(NioEndpoint.java:1468) > at > org.apache.tomcat.util.net.SocketWrapperBase$VectoredIOCompletionHandler.completed(SocketWrapperBase.java:1100) > at > org.apache.tomcat.util.net.NioEndpoint$NioSocketWrapper$NioOperationState.run(NioEndpoint.java:1511) > at > org.apache.tomcat.util.net.SocketWrapperBase$VectoredIOCompletionHandler.completed(SocketWrapperBase.java:1100) > ..... > ..... > ..... > ..... > at > org.apache.tomcat.util.net.NioEndpoint$NioSocketWrapper$NioOperationState.run(NioEndpoint.java:1511) > at > org.apache.tomcat.util.net.SocketWrapperBase$VectoredIOCompletionHandler.completed(SocketWrapperBase.java:1100) > at > org.apache.tomcat.util.net.NioEndpoint$NioSocketWrapper$NioOperationState.run(NioEndpoint.java:1511) > at > org.apache.tomcat.util.net.SocketWrapperBase$VectoredIOCompletionHandl > er.completed(SocketWrapperBase.java:1100) > > Is there anything that was fixed around this in latest 9.0.x branch ? Not obviously. I've reviewed every commit since c8ec2d4c. There is nothing that directly works with the I/O. There is 1e97ab2 which fixes a relatively recent regression in the HTTP/2 code. I guess it is possible (but it seems a bit of a stretch) that that bug is triggering an issue in JMeter which in turn is sending invalid HTTP/2 packets. I think at this point, given the relatively small number of commits between c8ec2d4c and HEAD, the most useful thing you could do is run a binary search to find out at which commit the issue is fixed. If we know which commit to look at that should help track down the root cause. Mark > > Thanks and Regards > Arshiya Shariff > > -----Original Message----- > From: Mark Thomas <ma...@apache.org> > Sent: Monday, October 5, 2020 9:52 PM > To: users@tomcat.apache.org > Subject: Re: HTTP2: memory filled up fast on increasing the > connections to 1000/2000 (Embedded tomcat 9.0.38) > > On 05/10/2020 10:56, Arshiya Shariff wrote: >> Hi All, >> >> Thank you so much Mark . >> We tested the jars built from latest 9.0.x with 2000 / 5000 users >> (connections) from JMeter , We see a very good improvement with the >> heap usage > > Good news. As is the fact that the other errors have been cleared up. > >> But I see this exception printed multiple times , I am not sure why this >> occurs : >> Exception in thread "http-nio-x.y.z-1234-exec-213" >> java.lang.StackOverflowError >> at sun.nio.ch.IOUtil.read(IOUtil.java:240) >> at sun.nio.ch.SocketChannelImpl.read(SocketChannelImpl.java:440) >> at org.apache.tomcat.util.net.NioChannel.read(NioChannel.java:174) >> at >> org.apache.tomcat.util.net.NioEndpoint$NioSocketWrapper$NioOperationState.run(NioEndpoint.java:1468) >> at >> org.apache.tomcat.util.net.SocketWrapperBase$VectoredIOCompletionHandler.completed(SocketWrapperBase.java:1100) >> at >> org.apache.tomcat.util.net.NioEndpoint$NioSocketWrapper$NioOperationState.run(NioEndpoint.java:1511) >> at >> org.apache.tomcat.util.net.SocketWrapperBase$VectoredIOCompletionHandler.completed(SocketWrapperBase.java:1100) >> at >> org.apache.tomcat.util.net.NioEndpoint$NioSocketWrapper$NioOperationState.run(NioEndpoint.java:1511) >> at >> org.apache.tomcat.util.net.SocketWrapperBase$VectoredIOCompletionHandler.completed(SocketWrapperBase.java:1100) >> at >> org.apache.tomcat.util.net.NioEndpoint$NioSocketWrapper$NioOperationS >> t >> ate.run(NioEndpoint.java:1511) > > That looks like an infinite loop reading an incoming frame. > New frames are read using a 9 byte buffer for the header and a 16k buffer for > the payload (since Tomcat sets this as the max frame size). > > The loop is occurring because one of those buffers is simultaneously both > full and still has more data to read. That should not be possible and I > haven't yet been able to figure out how this is happening. > > How easy is this to reproduce? > > How often do you see these errors in your test run? > > Do you have a reliable test case that reproduces this on a clean Tomcat 9.0.x > build? If is, can you share the details? > > Do you have the other end of that stack trace? I'm interested in how the code > enters the loop. > > Thanks, > > Mark > > --------------------------------------------------------------------- > To unsubscribe, e-mail: users-unsubscr...@tomcat.apache.org > For additional commands, e-mail: users-h...@tomcat.apache.org > > > --------------------------------------------------------------------- > To unsubscribe, e-mail: users-unsubscr...@tomcat.apache.org > For additional commands, e-mail: users-h...@tomcat.apache.org > --------------------------------------------------------------------- To unsubscribe, e-mail: users-unsubscr...@tomcat.apache.org For additional commands, e-mail: users-h...@tomcat.apache.org --------------------------------------------------------------------- To unsubscribe, e-mail: users-unsubscr...@tomcat.apache.org For additional commands, e-mail: users-h...@tomcat.apache.org
