On Thu, Aug 27, 2020 at 12:35 PM Christopher Schultz
<ch...@christopherschultz.net> wrote:
>
> -----BEGIN PGP SIGNED MESSAGE-----
> Hash: SHA256
>
> David,
>
> On 8/27/20 10:48, David wrote:
> > In the last two weeks I've had two occurrences where a single
> > CentOS 7 production server hosting a public webpage has become
> > unresponsive. The first time, all 300 available
> > "https-jsse-nio-8443" threads were consumed, with the max age being
> > around 45minutes, and all in a "S" status. This time all 300 were
> > consumed in "S" status with the oldest being around ~16minutes. A
> > restart of Tomcat on both occasions freed these threads and the
> > website became responsive again. The connections are post/get
> > methods which shouldn't take very long at all.
> >
> > CPU/MEM/JVM all appear to be within normal operating limits. I've
> > not had much luck searching for articles for this behavior nor
> > finding remedies. The default timeout values are used in both
> > Tomcat and in the applications that run within as far as I can
> > tell. Hopefully someone will have some insight on why the behavior
> > could be occurring, why isn't Tomcat killing the connections? Even
> > in a RST/ACK status, shouldn't Tomcat terminate the connection
> > without an ACK from the client after the default timeout?
>
> Can you please post:
>
> 1. Complete Tomcat version
I can't find anything more granular than 9.0.29, is there a command to
show a sub patch level?
> 2. Connector configuration (possibly redacted)
This is the 8443 section of the server.xml *8080 is available during
the outage and I'm able to curl the management page to see the 300
used threads, their status, and age*
<Service name="Catalina">
<!--The connectors can use a shared executor, you can define one
or more named thread pools-->
<!--
<Executor name="tomcatThreadPool" namePrefix="catalina-exec-"
maxThreads="150" minSpareThreads="4"/>
-->
<!-- A "Connector" represents an endpoint by which requests are received
and responses are returned. Documentation at :
Java HTTP Connector: /docs/config/http.html
Java AJP Connector: /docs/config/ajp.html
APR (HTTP/AJP) Connector: /docs/apr.html
Define a non-SSL/TLS HTTP/1.1 Connector on port 8080
-->
<Connector port="8080" protocol="HTTP/1.1"
connectionTimeout="20000"
redirectPort="8443" />
<!-- A "Connector" using the shared thread pool-->
<!--
<Connector executor="tomcatThreadPool"
port="8080" protocol="HTTP/1.1"
connectionTimeout="20000"
redirectPort="8443" />
-->
<!-- Define an SSL/TLS HTTP/1.1 Connector on port 8443
This connector uses the NIO implementation. The default
SSLImplementation will depend on the presence of the APR/native
library and the useOpenSSL attribute of the
AprLifecycleListener.
Either JSSE or OpenSSL style configuration may be used regardless of
the SSLImplementation selected. JSSE style configuration is used below.
-->
<Connector port="8443" protocol="org.apache.coyote.http11.Http11NioProtocol"
maxThreads="300" SSLEnabled="true" >
<SSLHostConfig>
<Certificate
certificateKeystoreFile="/opt/apache-tomcat-9.0.29/redacted.jks"
certificateKeystorePassword="redacted"
type="RSA" />
</SSLHostConfig>
</Connector>
<!-- Define an SSL/TLS HTTP/1.1 Connector on port 8443 with HTTP/2
This connector uses the APR/native implementation which always uses
OpenSSL for TLS.
Either JSSE or OpenSSL style configuration may be used. OpenSSL style
configuration is used below.
-->
<Connector port="8443" protocol="org.apache.coyote.http11.Http11NioProtocol"
maxThreads="300" SSLEnabled="true" >
<SSLHostConfig protocols="TLSv1.2">
<Certificate
certificateKeystoreFile="/opt/apache-tomcat-9.0.29/redacted.jks"
certificateKeystorePassword="redacted"
type="RSA" />
</SSLHostConfig>
</Connector>
>
> > Is there a graceful way to script the termination of threads in
> > case Tomcat isn't able to for whatever reason?
>
> Not really.
>
> > My research for killing threads results in system threads or
> > application threads, not Tomcat Connector connection threads, so
> > I'm not sure if this is even viable. I'm also looking into ways to
> > terminate these aged sessions via the F5. At this time I'm open to
> > any suggestions that would be able to automate a resolution to
> > keep the system from experiencing downtime, or for any insight on
> > where to look for a root cause. Thanks in advance for any guidance
> > you can lend.
> It might actually be the F5 itself, especially if it opens up a large
> number of connections to Tomcat and then tries to open additional ones
> for some reason. If it opens 300 connections (which are then e.g.
> leaked by the F5 internally) but the 301st is refused, then your
> server is essentially inert from that point forward.
>
> NIO connectors default to max 10k connections so that's not likely the
> actual problem, here, but it could be for some configurations.
>
> Do you have a single F5 or a group of them?
A group of them, several HA pairs depending on internal or external
and application. This server is behind one HA pair and is a single
server.
>
> - -chris
Thank you Chris!
David
> -----BEGIN PGP SIGNATURE-----
> Comment: Using GnuPG with Thunderbird - https://www.enigmail.net/
>
> iQIzBAEBCAAdFiEEMmKgYcQvxMe7tcJcHPApP6U8pFgFAl9H7tIACgkQHPApP6U8
> pFjR1hAAldbVnHDrV0W4aPLvcDwO/zi7qvrCscNjnJWhmR1m9TrevlrSb0EZvCJo
> gTl7DXYEiZ9sBEdgs6AavHlk8jQ+ZbXbp8lsMElW5X9QoxxUD3YyJEpDSeHOG7/S
> 2CyCYGzAQER0RlzBn9w97bCKWvUWoWDeLApd/pwdATjAo53hDtdNGdz6WdNLEzKm
> g/BCZP0ynHZu7pMzSeZsOUBUXEKhDwHU+71fJo+WIJ4Gtiyb4xf2qkewvjQtuOGl
> o/yESHNCJy09JAs3xK9W6eEVp981/Fuo4qDH32MJaXXbZRaNk32AwqngXKUhTW2l
> BBl0jHoFIj+YJYc6AgVlv0la5qDIqP2VTv4ujOLBeF/95oP4sVRobIN4TiFyH6vv
> ImvvRq55ML5NvKJv8g9Tj0aY5PusfwxcwyMCVovIof49vQXJUy7SbtgRB3eqgT8W
> WwdBiGNsyWZpVjpr/CGBkBZmuR4wckeq0J5O/XGRFS9pK1jXH4gPnxe58vJmjA+P
> RiSdp3SsU0P94SuF843CW+vmWyUu6SApCybUTwo5yiFXP2e/1+M9/fUGsykXpszU
> zUvMcj9LWJ1QR3TbvEnwilsge4HKbUM3ZsFaujDjAVy6TAOgGS/dtVZ2UyMcrlOd
> JMe3GeaOdM+ej27l5D8Eq6jaQcCfy+Mg9HxsYsbyrgrw3AhBhmo=
> =eVIu
> -----END PGP SIGNATURE-----
>
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: users-unsubscr...@tomcat.apache.org
> For additional commands, e-mail: users-h...@tomcat.apache.org
>
---------------------------------------------------------------------
To unsubscribe, e-mail: users-unsubscr...@tomcat.apache.org
For additional commands, e-mail: users-h...@tomcat.apache.org
