On 11/9/2020 11:05 AM, Jerry Malcolm wrote:
On 11/9/2020 3:10 AM, Mark Thomas wrote:
On 08/11/2020 01:33, Jerry Malcolm wrote:
On 11/7/2020 6:56 PM, Christopher Schultz wrote:
Jerry,
On 11/6/20 19:49, Jerry Malcolm wrote:
I have a relatively new environment with a standalone tomcat (8.5)
running on an AWS Linux2 EC2. I'm not using HTTPD/AJP. Its a direct
connection to port 443. (Well technically, I have firewallD in the
flow in order to route the protected port 80 to port 8080 and 443 to
8443 for TC).
I am doing some stress testing on the server and failing miserably.
I am sending around 130 ajax calls in rapid succession using HTTP/2.
These are all very simple small page (JSP) requests. Not a lot of
processing required. The first ~75 requests process normally. Then
everything hangs up. In the tomcat logs I'm getting a bunch of
"Timeout waiting to read data from client" exceptions. And in the
stacktrace for these exceptions, they are all occurring when I'm
trying to access a parameter from the request. Looking at the
request network timing in the browser console, I see a bunch of the
requests returning in typical time of a few milliseconds. Then
another large block of requests that all start returning around 4
seconds, then another block that wait until 8 seconds to return.
I've tried firefox and chrome with the same results.
I've been using httpd in front of TC for years. So this is the first
time I'm running TC standalone. It is very likely I've got some
parameters set horribly wrong. But I have no clue where to start.
This is not a tiny EC2, and my internet connection is not showing any
signs of problems. So I really don't think this is a
performance-related problem. The problem is very consistent and
reproducible with the same counts of success/failure calls. What
could be causing the "Timeout waiting to read data from client" after
75 calls, and then cause blocks of calls to wait 4 seconds, 8
seconds, etc before responding? I really need to handle more
simultaneous load that this is currently allowing.
Thanks in advance for the education.
Are you using HTTP Keepalives on your connections? Are you actually
re-using those connections in your test? What is your keepalive
timeout on your <Connector>. Actually, what is your whole <Connector>
configuration?
-chris
Hi Chris, here are my two connector definitions from server.xml:
<Connector
port="8080"
protocol="HTTP/1.1"
connectionTimeout="20000"
redirectPort="443" />
<Connector
port="8443"
maxThreads="150"
connectionTimeout="20000"
SSLEnabled="true"
scheme="https"
secure="true"
clientAuth="false"
SSLCertificateFile="ssl/aaaaa.com/cert.pem"
SSLCertificateChainFile="ssl/aaaaa.com/chain.pem"
SSLCertificateKeyFile="ssl/aaaaa.com/privkey.pem">
<UpgradeProtocol
className="org.apache.coyote.http2.Http2Protocol" />
</Connector>
How are you stress testing this? All on a single HTTP/2 connection or
multiple connections? With which tool?
You might want to test HTTP/1.1 requests (with and without TLS) to see
if the problem is specific to HTTP/2 or TLS as that should help narrow
down the root cause.
Mark
Hi Mark, technically it's not a 'designed' stress test. It's real
production code that just happens to stress the server more than
usual. It's just a page that makes a bunch of ajax calls, and the
responses to each of those issue a second ajax call.
If you don't see anything obvious in my configuration, we will
definitely pursue the http/1.1 options, etc. I just wanted to
eliminate the chance of obvious 'pilot error' before digging deeper.
Specifically, where is that error detected in the TC flow? In my logs
it fails on getting request parameters. It sounds like the input
reader for the request is getting blocked. But the first part of
the request is getting in since it does route to the appropriate JSP.
Just seems strange that the http/2 or ssl layers would let half of the
request in and then block the rest of the request. The browser
appears to be sending everything. And it fails the same using firefox
or chrome. Any ideas?
Thx
Update on this. One of our clients got ERR_HTTP2_SERVER_REFUSED_STREAM
after things locked up. I removed the http2 'upgrade protocol' line
from my connector, and everything works. So it's apparently something
wrong with my http2 setup. Ideas? (See my connector config above in
this thread).
Thx
