Hi,
Mark , with the test runs that I performed over clean 9.0.x branch I was not
able to reproduce this. But with 9.0.38 and the jars built from 9.0.x with
hash: c8ec2d4cde3a31b0e9df9a30e7915d77ba725545 , with 700 or 1000 users
(connections) and on sending 1000 Requests per second (or even lesser) ,
payload of 16K from JMeter I can see that this Exception occurs within few
minutes of starting the test . The maxThreads configured in tomcat is 200 .
How often do you see these errors in your test run?
Randomly, at times 2 or 3 such traces.
Do you have the other end of that stack trace?
It is only the two lines that is recursively printed till the end about ~500
times in one trace :
at
org.apache.tomcat.util.net.NioEndpoint$NioSocketWrapper$NioOperationState.run(NioEndpoint.java:1511)
at
org.apache.tomcat.util.net.SocketWrapperBase$VectoredIOCompletionHandler.completed(SocketWrapperBase.java:1100)
I see the trace starting with :
Exception in thread "http-nio-x.y.z-1090-exec-107" java.lang.StackOverflowError
at sun.nio.ch.SocketChannelImpl.read(SocketChannelImpl.java:446)
at org.apache.tomcat.util.net.NioChannel.read(NioChannel.java:174)
at
org.apache.tomcat.util.net.NioEndpoint$NioSocketWrapper$NioOperationState.run(NioEndpoint.java:1468)
at
org.apache.tomcat.util.net.SocketWrapperBase$VectoredIOCompletionHandler.completed(SocketWrapperBase.java:1100)
(OR)
Exception in thread "http-nio-x.y.z-1090-exec-87" java.lang.StackOverflowError
at sun.nio.ch.IOVecWrapper.get(IOVecWrapper.java:96)
at sun.nio.ch.IOUtil.read(IOUtil.java:240)
at sun.nio.ch.SocketChannelImpl.read(SocketChannelImpl.java:440)
at org.apache.tomcat.util.net.NioChannel.read(NioChannel.java:174)
at
org.apache.tomcat.util.net.NioEndpoint$NioSocketWrapper$NioOperationState.run(NioEndpoint.java:1468)
at
org.apache.tomcat.util.net.SocketWrapperBase$VectoredIOCompletionHandler.completed(SocketWrapperBase.java:1100)
at
org.apache.tomcat.util.net.NioEndpoint$NioSocketWrapper$NioOperationState.run(NioEndpoint.java:1511)
at
org.apache.tomcat.util.net.SocketWrapperBase$VectoredIOCompletionHandler.completed(SocketWrapperBase.java:1100)
.....
.....
.....
.....
at
org.apache.tomcat.util.net.NioEndpoint$NioSocketWrapper$NioOperationState.run(NioEndpoint.java:1511)
at
org.apache.tomcat.util.net.SocketWrapperBase$VectoredIOCompletionHandler.completed(SocketWrapperBase.java:1100)
at
org.apache.tomcat.util.net.NioEndpoint$NioSocketWrapper$NioOperationState.run(NioEndpoint.java:1511)
at
org.apache.tomcat.util.net.SocketWrapperBase$VectoredIOCompletionHandler.completed(SocketWrapperBase.java:1100)
Is there anything that was fixed around this in latest 9.0.x branch ?
Thanks and Regards
Arshiya Shariff
-----Original Message-----
From: Mark Thomas <ma...@apache.org>
Sent: Monday, October 5, 2020 9:52 PM
To: users@tomcat.apache.org
Subject: Re: HTTP2: memory filled up fast on increasing the connections to
1000/2000 (Embedded tomcat 9.0.38)
On 05/10/2020 10:56, Arshiya Shariff wrote:
> Hi All,
>
> Thank you so much Mark .
> We tested the jars built from latest 9.0.x with 2000 / 5000 users
> (connections) from JMeter , We see a very good improvement with the
> heap usage
Good news. As is the fact that the other errors have been cleared up.
> But I see this exception printed multiple times , I am not sure why this
> occurs :
> Exception in thread "http-nio-x.y.z-1234-exec-213"
> java.lang.StackOverflowError
> at sun.nio.ch.IOUtil.read(IOUtil.java:240)
> at sun.nio.ch.SocketChannelImpl.read(SocketChannelImpl.java:440)
> at org.apache.tomcat.util.net.NioChannel.read(NioChannel.java:174)
> at
> org.apache.tomcat.util.net.NioEndpoint$NioSocketWrapper$NioOperationState.run(NioEndpoint.java:1468)
> at
> org.apache.tomcat.util.net.SocketWrapperBase$VectoredIOCompletionHandler.completed(SocketWrapperBase.java:1100)
> at
> org.apache.tomcat.util.net.NioEndpoint$NioSocketWrapper$NioOperationState.run(NioEndpoint.java:1511)
> at
> org.apache.tomcat.util.net.SocketWrapperBase$VectoredIOCompletionHandler.completed(SocketWrapperBase.java:1100)
> at
> org.apache.tomcat.util.net.NioEndpoint$NioSocketWrapper$NioOperationState.run(NioEndpoint.java:1511)
> at
> org.apache.tomcat.util.net.SocketWrapperBase$VectoredIOCompletionHandler.completed(SocketWrapperBase.java:1100)
> at
> org.apache.tomcat.util.net.NioEndpoint$NioSocketWrapper$NioOperationSt
> ate.run(NioEndpoint.java:1511)
That looks like an infinite loop reading an incoming frame.
New frames are read using a 9 byte buffer for the header and a 16k buffer for
the payload (since Tomcat sets this as the max frame size).
The loop is occurring because one of those buffers is simultaneously both full
and still has more data to read. That should not be possible and I haven't yet
been able to figure out how this is happening.
How easy is this to reproduce?
How often do you see these errors in your test run?
Do you have a reliable test case that reproduces this on a clean Tomcat 9.0.x
build? If is, can you share the details?
Do you have the other end of that stack trace? I'm interested in how the code
enters the loop.
Thanks,
Mark
---------------------------------------------------------------------
To unsubscribe, e-mail: users-unsubscr...@tomcat.apache.org
For additional commands, e-mail: users-h...@tomcat.apache.org
