Thank you all for the replies! On Thu, Aug 27, 2020 at 3:53 PM Christopher Schultz <ch...@christopherschultz.net> wrote: > > -----BEGIN PGP SIGNED MESSAGE----- > Hash: SHA256 > > David, > > On 8/27/20 13:57, David wrote: > > On Thu, Aug 27, 2020 at 12:35 PM Christopher Schultz > > <ch...@christopherschultz.net> wrote: > >> > > David, > > > > On 8/27/20 10:48, David wrote: > >>>> In the last two weeks I've had two occurrences where a > >>>> single CentOS 7 production server hosting a public webpage > >>>> has become unresponsive. The first time, all 300 available > >>>> "https-jsse-nio-8443" threads were consumed, with the max age > >>>> being around 45minutes, and all in a "S" status. This time > >>>> all 300 were consumed in "S" status with the oldest being > >>>> around ~16minutes. A restart of Tomcat on both occasions > >>>> freed these threads and the website became responsive again. > >>>> The connections are post/get methods which shouldn't take > >>>> very long at all. > >>>> > >>>> CPU/MEM/JVM all appear to be within normal operating limits. > >>>> I've not had much luck searching for articles for this > >>>> behavior nor finding remedies. The default timeout values are > >>>> used in both Tomcat and in the applications that run within > >>>> as far as I can tell. Hopefully someone will have some > >>>> insight on why the behavior could be occurring, why isn't > >>>> Tomcat killing the connections? Even in a RST/ACK status, > >>>> shouldn't Tomcat terminate the connection without an ACK from > >>>> the client after the default timeout? > > > > Can you please post: > > > > 1. Complete Tomcat version > >> I can't find anything more granular than 9.0.29, is there a > >> command to show a sub patch level? > > 9.0.29 is the patch-level, so that's fine. You are about 10 versions > out of date (~1 year). Any chance for an upgrade?
They had to re-dev many apps last year when we upgraded from I want to say 1 or 3 or something equally as horrific. Hopefully they are forward compatible with the newer releases and if not should surely be tackled now before later, I will certainly bring this to the table! > > > 2. Connector configuration (possibly redacted) > >> This is the 8443 section of the server.xml *8080 is available > >> during the outage and I'm able to curl the management page to see > >> the 300 used threads, their status, and age* <Service > >> name="Catalina"> > >> > >> [snip] > >> > >> <Connector port="8080" protocol="HTTP/1.1" > >> connectionTimeout="20000" redirectPort="8443" /> [snip] > >> <Connector port="8443" > >> protocol="org.apache.coyote.http11.Http11NioProtocol" > >> maxThreads="300" SSLEnabled="true" > <SSLHostConfig> > >> <Certificate > >> certificateKeystoreFile="/opt/apache-tomcat-9.0.29/redacted.jks" > >> certificateKeystorePassword="redacted" type="RSA" /> > >> </SSLHostConfig> </Connector> [snip] <Connector port="8443" > >> protocol="org.apache.coyote.http11.Http11NioProtocol" > >> maxThreads="300" SSLEnabled="true" > <SSLHostConfig > >> protocols="TLSv1.2"> <Certificate > >> certificateKeystoreFile="/opt/apache-tomcat-9.0.29/redacted.jks" > >> certificateKeystorePassword="redacted" type="RSA" /> > >> </SSLHostConfig> </Connector> > > What, two connectors on one port? Do you get errors when starting? No errors, one is "with HTTP2" should I delete the other former? > > I don't see anything obviously problematic in the above configuration > (other than the double-definition of the 8443 connector). > > 300 tied-up connections (from your initial report) sounds like a > significant number: probably the thread count. Yes sir, that's the NIO thread count for the 8443 connector. > > Mark (as is often the case) is right: take some thread dumps next time > everything locks up and see what all those threads are doing. Often, > it's something like everything is awaiting on a db connection and the > db pool has been exhausted or something. Relatively simple quick-fixes > are available for that, and better, longer-term fixes as well. > Mark/Chris is there a way to dump the connector threads specifically? Or simply is it all contained as a machine/process thread? Sorry I'm not really a Linux guy. > > Do you have a single F5 or a group of them? > >> A group of them, several HA pairs depending on internal or > >> external and application. This server is behind one HA pair and > >> is a single server. > > Okay. Just remember that each F5 can make some large number of > connections to Tomcat, so you need to make sure you can handle them. > > This was a much bigger deal back in the BIO days when thread limit = > connection limit, and the thread limit was usually something like 250 > - - 300. NIO is much better, and the default connection limit is 10k > which "ought to be enough for anyone"[1]. (lol) I'm more used to the 1-1 of the BIO style, which kinda confused me when I asked the F5 to truncate >X connections and alert me and there were 600+ connections while Tomcat manager stated ~30. Then I read what the non-interrupt was about. > > > > [1] With apologies to Bill gates, who apparently never said anything > of the sort. Thanks again, David > -----BEGIN PGP SIGNATURE----- > Comment: Using GnuPG with Thunderbird - https://www.enigmail.net/ > > iQIzBAEBCAAdFiEEMmKgYcQvxMe7tcJcHPApP6U8pFgFAl9IHUYACgkQHPApP6U8 > pFgcMhAAsN/Fc0nG4EJ/aaxZtj46g7FW2UDLa3HcGI+r8mvI5pYlCxWO0Cm4oDHn > PAEUsjNgDcyLbWPa+hIfTWZ2v594w8ACrprpdNWHoPhZ316LudpG3G8RWwrIVsOa > pn6MmX79rvds1Htl2cbsZkaaNCg/3+dx5VgyQtexHopSP9FpU1swDwex4fIf/pFz > jcl4SB6eMnKxHwf/avwEy6sfdN05ALCl6KfJBCA6vnRvMT8hYVGs5B/bDdPRU5zL > 0cNIAlNaxrcw0G13cuOhg5KYG+eeKBKl2R/OiSmyn4+Xp7zzbl3G3i4GvfbYrwqe > BFTcTGT3cTE3vwMcHmsskh2soxYcH3etWtJ2/XsrKoKdRqXpWybVyNEvHcUwhxdP > h7SAN5V8D2+9a8Hhh8y/hUCHBOT70THUyBipYweV26wUj4ipOAiYiJ2UaCATwNzf > E7Giv6D4Y9WQCU119HaQ65TLmvGTtfzctM5pJzrnRbI7LOpuo9/bNYxkYNoU8U9r > sAgY4t3kvKNttetFnwdY5+JTM+yrFolYFkYMFv8vpaVyiumP4+dpbkniRAmLabWl > O0kIn/bRTkek4ic/qCuawBi1Rc1hV1/1uUE1+t8XHG7Sfdd0vwUabZ8ZRxNUBWcc > EliCfzyMosWcsgU2puNduPyXDeRxKb5gfe4VdfaH5xvfdqIpfgw= > =SesB > -----END PGP SIGNATURE----- > > --------------------------------------------------------------------- > To unsubscribe, e-mail: users-unsubscr...@tomcat.apache.org > For additional commands, e-mail: users-h...@tomcat.apache.org > --------------------------------------------------------------------- To unsubscribe, e-mail: users-unsubscr...@tomcat.apache.org For additional commands, e-mail: users-h...@tomcat.apache.org
