-----BEGIN PGP SIGNED MESSAGE----- Hash: SHA256 David,
On 8/27/20 10:48, David wrote: > In the last two weeks I've had two occurrences where a single > CentOS 7 production server hosting a public webpage has become > unresponsive. The first time, all 300 available > "https-jsse-nio-8443" threads were consumed, with the max age being > around 45minutes, and all in a "S" status. This time all 300 were > consumed in "S" status with the oldest being around ~16minutes. A > restart of Tomcat on both occasions freed these threads and the > website became responsive again. The connections are post/get > methods which shouldn't take very long at all. > > CPU/MEM/JVM all appear to be within normal operating limits. I've > not had much luck searching for articles for this behavior nor > finding remedies. The default timeout values are used in both > Tomcat and in the applications that run within as far as I can > tell. Hopefully someone will have some insight on why the behavior > could be occurring, why isn't Tomcat killing the connections? Even > in a RST/ACK status, shouldn't Tomcat terminate the connection > without an ACK from the client after the default timeout? Can you please post: 1. Complete Tomcat version 2. Connector configuration (possibly redacted) > Is there a graceful way to script the termination of threads in > case Tomcat isn't able to for whatever reason? Not really. > My research for killing threads results in system threads or > application threads, not Tomcat Connector connection threads, so > I'm not sure if this is even viable. I'm also looking into ways to > terminate these aged sessions via the F5. At this time I'm open to > any suggestions that would be able to automate a resolution to > keep the system from experiencing downtime, or for any insight on > where to look for a root cause. Thanks in advance for any guidance > you can lend. It might actually be the F5 itself, especially if it opens up a large number of connections to Tomcat and then tries to open additional ones for some reason. If it opens 300 connections (which are then e.g. leaked by the F5 internally) but the 301st is refused, then your server is essentially inert from that point forward. NIO connectors default to max 10k connections so that's not likely the actual problem, here, but it could be for some configurations. Do you have a single F5 or a group of them? - -chris -----BEGIN PGP SIGNATURE----- Comment: Using GnuPG with Thunderbird - https://www.enigmail.net/ iQIzBAEBCAAdFiEEMmKgYcQvxMe7tcJcHPApP6U8pFgFAl9H7tIACgkQHPApP6U8 pFjR1hAAldbVnHDrV0W4aPLvcDwO/zi7qvrCscNjnJWhmR1m9TrevlrSb0EZvCJo gTl7DXYEiZ9sBEdgs6AavHlk8jQ+ZbXbp8lsMElW5X9QoxxUD3YyJEpDSeHOG7/S 2CyCYGzAQER0RlzBn9w97bCKWvUWoWDeLApd/pwdATjAo53hDtdNGdz6WdNLEzKm g/BCZP0ynHZu7pMzSeZsOUBUXEKhDwHU+71fJo+WIJ4Gtiyb4xf2qkewvjQtuOGl o/yESHNCJy09JAs3xK9W6eEVp981/Fuo4qDH32MJaXXbZRaNk32AwqngXKUhTW2l BBl0jHoFIj+YJYc6AgVlv0la5qDIqP2VTv4ujOLBeF/95oP4sVRobIN4TiFyH6vv ImvvRq55ML5NvKJv8g9Tj0aY5PusfwxcwyMCVovIof49vQXJUy7SbtgRB3eqgT8W WwdBiGNsyWZpVjpr/CGBkBZmuR4wckeq0J5O/XGRFS9pK1jXH4gPnxe58vJmjA+P RiSdp3SsU0P94SuF843CW+vmWyUu6SApCybUTwo5yiFXP2e/1+M9/fUGsykXpszU zUvMcj9LWJ1QR3TbvEnwilsge4HKbUM3ZsFaujDjAVy6TAOgGS/dtVZ2UyMcrlOd JMe3GeaOdM+ej27l5D8Eq6jaQcCfy+Mg9HxsYsbyrgrw3AhBhmo= =eVIu -----END PGP SIGNATURE----- --------------------------------------------------------------------- To unsubscribe, e-mail: users-unsubscr...@tomcat.apache.org For additional commands, e-mail: users-h...@tomcat.apache.org
