Hi Arshiya,
On Fri, Oct 9, 2020 at 2:33 PM Arshiya Shariff <arshiya.shar...@ericsson.com.invalid> wrote: > Hi, > > Mark , with the test runs that I performed over clean 9.0.x branch I was > not able to reproduce this. But with 9.0.38 and the jars built from 9.0.x > with hash: c8ec2d4cde3a31b0e9df9a30e7915d77ba725545 , with 700 or 1000 > users (connections) and on sending 1000 Requests per second (or even > lesser) , payload of 16K from JMeter I can see that this Exception occurs > within few minutes of starting the test . The maxThreads configured in > tomcat is 200 . > > How often do you see these errors in your test run? > Randomly, at times 2 or 3 such traces. > > Do you have the other end of that stack trace? > It is only the two lines that is recursively printed till the end about > ~500 times in one trace : > at org.apache.tomcat.util.net > .NioEndpoint$NioSocketWrapper$NioOperationState.run(NioEndpoint.java:1511) > at org.apache.tomcat.util.net > .SocketWrapperBase$VectoredIOCompletionHandler.completed(SocketWrapperBase.java:1100) > > I see the trace starting with : > Exception in thread "http-nio-x.y.z-1090-exec-107" > java.lang.StackOverflowError > at sun.nio.ch.SocketChannelImpl.read(SocketChannelImpl.java:446) > at org.apache.tomcat.util.net.NioChannel.read(NioChannel.java:174) > at org.apache.tomcat.util.net > .NioEndpoint$NioSocketWrapper$NioOperationState.run(NioEndpoint.java:1468) > at org.apache.tomcat.util.net > .SocketWrapperBase$VectoredIOCompletionHandler.completed(SocketWrapperBase.java:1100) > > (OR) > > Exception in thread "http-nio-x.y.z-1090-exec-87" > java.lang.StackOverflowError > at sun.nio.ch.IOVecWrapper.get(IOVecWrapper.java:96) > at sun.nio.ch.IOUtil.read(IOUtil.java:240) > at sun.nio.ch.SocketChannelImpl.read(SocketChannelImpl.java:440) > at org.apache.tomcat.util.net.NioChannel.read(NioChannel.java:174) > at org.apache.tomcat.util.net > .NioEndpoint$NioSocketWrapper$NioOperationState.run(NioEndpoint.java:1468) > at org.apache.tomcat.util.net > .SocketWrapperBase$VectoredIOCompletionHandler.completed(SocketWrapperBase.java:1100) > at org.apache.tomcat.util.net > .NioEndpoint$NioSocketWrapper$NioOperationState.run(NioEndpoint.java:1511) > at org.apache.tomcat.util.net > .SocketWrapperBase$VectoredIOCompletionHandler.completed(SocketWrapperBase.java:1100) > ..... > ..... > ..... > ..... > at org.apache.tomcat.util.net > .NioEndpoint$NioSocketWrapper$NioOperationState.run(NioEndpoint.java:1511) > at org.apache.tomcat.util.net > .SocketWrapperBase$VectoredIOCompletionHandler.completed(SocketWrapperBase.java:1100) > at org.apache.tomcat.util.net > .NioEndpoint$NioSocketWrapper$NioOperationState.run(NioEndpoint.java:1511) > at org.apache.tomcat.util.net > .SocketWrapperBase$VectoredIOCompletionHandler.completed(SocketWrapperBase.java:1100) > > Is there anything that was fixed around this in latest 9.0.x branch ? > 9.0.39 is being voted now at dev@ mailing list: It can be obtained from: https://dist.apache.org/repos/dist/dev/tomcat/tomcat-9/v9.0.39/ The Maven staging repo is: https://repository.apache.org/content/repositories/orgapachetomcat-1281/ Give it a try and vote! > Thanks and Regards > Arshiya Shariff > > -----Original Message----- > From: Mark Thomas <ma...@apache.org> > Sent: Monday, October 5, 2020 9:52 PM > To: users@tomcat.apache.org > Subject: Re: HTTP2: memory filled up fast on increasing the connections to > 1000/2000 (Embedded tomcat 9.0.38) > > On 05/10/2020 10:56, Arshiya Shariff wrote: > > Hi All, > > > > Thank you so much Mark . > > We tested the jars built from latest 9.0.x with 2000 / 5000 users > > (connections) from JMeter , We see a very good improvement with the > > heap usage > > Good news. As is the fact that the other errors have been cleared up. > > > But I see this exception printed multiple times , I am not sure why this > occurs : > > Exception in thread "http-nio-x.y.z-1234-exec-213" > java.lang.StackOverflowError > > at sun.nio.ch.IOUtil.read(IOUtil.java:240) > > at sun.nio.ch.SocketChannelImpl.read(SocketChannelImpl.java:440) > > at org.apache.tomcat.util.net > .NioChannel.read(NioChannel.java:174) > > at org.apache.tomcat.util.net > .NioEndpoint$NioSocketWrapper$NioOperationState.run(NioEndpoint.java:1468) > > at org.apache.tomcat.util.net > .SocketWrapperBase$VectoredIOCompletionHandler.completed(SocketWrapperBase.java:1100) > > at org.apache.tomcat.util.net > .NioEndpoint$NioSocketWrapper$NioOperationState.run(NioEndpoint.java:1511) > > at org.apache.tomcat.util.net > .SocketWrapperBase$VectoredIOCompletionHandler.completed(SocketWrapperBase.java:1100) > > at org.apache.tomcat.util.net > .NioEndpoint$NioSocketWrapper$NioOperationState.run(NioEndpoint.java:1511) > > at org.apache.tomcat.util.net > .SocketWrapperBase$VectoredIOCompletionHandler.completed(SocketWrapperBase.java:1100) > > at > > org.apache.tomcat.util.net.NioEndpoint$NioSocketWrapper$NioOperationSt > > ate.run(NioEndpoint.java:1511) > > That looks like an infinite loop reading an incoming frame. > New frames are read using a 9 byte buffer for the header and a 16k buffer > for the payload (since Tomcat sets this as the max frame size). > > The loop is occurring because one of those buffers is simultaneously both > full and still has more data to read. That should not be possible and I > haven't yet been able to figure out how this is happening. > > How easy is this to reproduce? > > How often do you see these errors in your test run? > > Do you have a reliable test case that reproduces this on a clean Tomcat > 9.0.x build? If is, can you share the details? > > Do you have the other end of that stack trace? I'm interested in how the > code enters the loop. > > Thanks, > > Mark > > --------------------------------------------------------------------- > To unsubscribe, e-mail: users-unsubscr...@tomcat.apache.org > For additional commands, e-mail: users-h...@tomcat.apache.org > >
