-----BEGIN PGP SIGNED MESSAGE----- Hash: SHA256 David,
On 8/27/20 13:57, David wrote: > On Thu, Aug 27, 2020 at 12:35 PM Christopher Schultz > <ch...@christopherschultz.net> wrote: >> > David, > > On 8/27/20 10:48, David wrote: >>>> In the last two weeks I've had two occurrences where a >>>> single CentOS 7 production server hosting a public webpage >>>> has become unresponsive. The first time, all 300 available >>>> "https-jsse-nio-8443" threads were consumed, with the max age >>>> being around 45minutes, and all in a "S" status. This time >>>> all 300 were consumed in "S" status with the oldest being >>>> around ~16minutes. A restart of Tomcat on both occasions >>>> freed these threads and the website became responsive again. >>>> The connections are post/get methods which shouldn't take >>>> very long at all. >>>> >>>> CPU/MEM/JVM all appear to be within normal operating limits. >>>> I've not had much luck searching for articles for this >>>> behavior nor finding remedies. The default timeout values are >>>> used in both Tomcat and in the applications that run within >>>> as far as I can tell. Hopefully someone will have some >>>> insight on why the behavior could be occurring, why isn't >>>> Tomcat killing the connections? Even in a RST/ACK status, >>>> shouldn't Tomcat terminate the connection without an ACK from >>>> the client after the default timeout? > > Can you please post: > > 1. Complete Tomcat version >> I can't find anything more granular than 9.0.29, is there a >> command to show a sub patch level? 9.0.29 is the patch-level, so that's fine. You are about 10 versions out of date (~1 year). Any chance for an upgrade? > 2. Connector configuration (possibly redacted) >> This is the 8443 section of the server.xml *8080 is available >> during the outage and I'm able to curl the management page to see >> the 300 used threads, their status, and age* <Service >> name="Catalina"> >> >> [snip] >> >> <Connector port="8080" protocol="HTTP/1.1" >> connectionTimeout="20000" redirectPort="8443" /> [snip] >> <Connector port="8443" >> protocol="org.apache.coyote.http11.Http11NioProtocol" >> maxThreads="300" SSLEnabled="true" > <SSLHostConfig> >> <Certificate >> certificateKeystoreFile="/opt/apache-tomcat-9.0.29/redacted.jks" >> certificateKeystorePassword="redacted" type="RSA" /> >> </SSLHostConfig> </Connector> [snip] <Connector port="8443" >> protocol="org.apache.coyote.http11.Http11NioProtocol" >> maxThreads="300" SSLEnabled="true" > <SSLHostConfig >> protocols="TLSv1.2"> <Certificate >> certificateKeystoreFile="/opt/apache-tomcat-9.0.29/redacted.jks" >> certificateKeystorePassword="redacted" type="RSA" /> >> </SSLHostConfig> </Connector> What, two connectors on one port? Do you get errors when starting? I don't see anything obviously problematic in the above configuration (other than the double-definition of the 8443 connector). 300 tied-up connections (from your initial report) sounds like a significant number: probably the thread count. Mark (as is often the case) is right: take some thread dumps next time everything locks up and see what all those threads are doing. Often, it's something like everything is awaiting on a db connection and the db pool has been exhausted or something. Relatively simple quick-fixes are available for that, and better, longer-term fixes as well. > Do you have a single F5 or a group of them? >> A group of them, several HA pairs depending on internal or >> external and application. This server is behind one HA pair and >> is a single server. Okay. Just remember that each F5 can make some large number of connections to Tomcat, so you need to make sure you can handle them. This was a much bigger deal back in the BIO days when thread limit = connection limit, and the thread limit was usually something like 250 - - 300. NIO is much better, and the default connection limit is 10k which "ought to be enough for anyone"[1]. - -chris [1] With apologies to Bill gates, who apparently never said anything of the sort. -----BEGIN PGP SIGNATURE----- Comment: Using GnuPG with Thunderbird - https://www.enigmail.net/ iQIzBAEBCAAdFiEEMmKgYcQvxMe7tcJcHPApP6U8pFgFAl9IHUYACgkQHPApP6U8 pFgcMhAAsN/Fc0nG4EJ/aaxZtj46g7FW2UDLa3HcGI+r8mvI5pYlCxWO0Cm4oDHn PAEUsjNgDcyLbWPa+hIfTWZ2v594w8ACrprpdNWHoPhZ316LudpG3G8RWwrIVsOa pn6MmX79rvds1Htl2cbsZkaaNCg/3+dx5VgyQtexHopSP9FpU1swDwex4fIf/pFz jcl4SB6eMnKxHwf/avwEy6sfdN05ALCl6KfJBCA6vnRvMT8hYVGs5B/bDdPRU5zL 0cNIAlNaxrcw0G13cuOhg5KYG+eeKBKl2R/OiSmyn4+Xp7zzbl3G3i4GvfbYrwqe BFTcTGT3cTE3vwMcHmsskh2soxYcH3etWtJ2/XsrKoKdRqXpWybVyNEvHcUwhxdP h7SAN5V8D2+9a8Hhh8y/hUCHBOT70THUyBipYweV26wUj4ipOAiYiJ2UaCATwNzf E7Giv6D4Y9WQCU119HaQ65TLmvGTtfzctM5pJzrnRbI7LOpuo9/bNYxkYNoU8U9r sAgY4t3kvKNttetFnwdY5+JTM+yrFolYFkYMFv8vpaVyiumP4+dpbkniRAmLabWl O0kIn/bRTkek4ic/qCuawBi1Rc1hV1/1uUE1+t8XHG7Sfdd0vwUabZ8ZRxNUBWcc EliCfzyMosWcsgU2puNduPyXDeRxKb5gfe4VdfaH5xvfdqIpfgw= =SesB -----END PGP SIGNATURE----- --------------------------------------------------------------------- To unsubscribe, e-mail: users-unsubscr...@tomcat.apache.org For additional commands, e-mail: users-h...@tomcat.apache.org
