-----BEGIN PGP SIGNED MESSAGE----- Hash: SHA256 Felix,
On 8/27/20 16:09, Felix Schumacher wrote: > > Am 27.08.20 um 19:35 schrieb Christopher Schultz: >> David, >> >> On 8/27/20 10:48, David wrote: >>> In the last two weeks I've had two occurrences where a single >>> CentOS 7 production server hosting a public webpage has become >>> unresponsive. The first time, all 300 available >>> "https-jsse-nio-8443" threads were consumed, with the max age >>> being around 45minutes, and all in a "S" status. This time all >>> 300 were consumed in "S" status with the oldest being around >>> ~16minutes. A restart of Tomcat on both occasions freed these >>> threads and the website became responsive again. The >>> connections are post/get methods which shouldn't take very long >>> at all. >> >>> CPU/MEM/JVM all appear to be within normal operating limits. >>> I've not had much luck searching for articles for this behavior >>> nor finding remedies. The default timeout values are used in >>> both Tomcat and in the applications that run within as far as I >>> can tell. Hopefully someone will have some insight on why the >>> behavior could be occurring, why isn't Tomcat killing the >>> connections? Even in a RST/ACK status, shouldn't Tomcat >>> terminate the connection without an ACK from the client after >>> the default timeout? >> >> Can you please post: >> >> 1. Complete Tomcat version 2. Connector configuration (possibly >> redacted) >> >>> Is there a graceful way to script the termination of threads >>> in case Tomcat isn't able to for whatever reason? >> >> Not really. > > (First look at Marks response on determining the root cause) > > Well, there might be a way (if it is sane, I don't know). You can > configure a valve to look for seemingly stuck threads and try to > interrupt them: > > http://tomcat.apache.org/tomcat-9.0-doc/config/valve.html#Stuck_Thread _Detection_Valve > > There are a few caveats there. First it is only working, when > both conditions are true > > * the servlets are synchronous * the stuck thread can be "freed" > with an Interrupt > > But really, if your threads are stuck for more than 15 minutes, you > have ample of time to take a thread dump and hopefully find the > root cause, so that you don't need this valve. This is a good idea as a band-aid, but the reality is that if you need the StuckThreadDetectionValve then your application is probably broken and needs to be fixed. Here are things that can be broken which might cause thread exhaustion: 1. Poor resource management. Things like db connections pools which can leak and/or not be refilled by the application. Everything stops when the db pool dries up. 2. Failure to set proper IO timeouts. Guess what the default read timeout is on a socket? Forever! If you read from a socket you might never hear back. Sounds like a problem. Set your read timeouts, kids. You might need to do this on your HTTP connections (and pools, and factories, and connection-wrappers like Apache http-client), your database config (usually in the config URL), and any remote-API libraries you are using (which use e.g. HTTP under the hood). - -chris -----BEGIN PGP SIGNATURE----- Comment: Using GnuPG with Thunderbird - https://www.enigmail.net/ iQIzBAEBCAAdFiEEMmKgYcQvxMe7tcJcHPApP6U8pFgFAl9IICoACgkQHPApP6U8 pFgkuQ/+NE7tC+wWXoP2Ntv0yljJHyasRY/3dVewoNUxfO4CwcEkhSpK5YEkiHd3 sE7jygxEn3SHtHJ0WQPBWMAzL9RoLnglbAsxVXuWCzbQzd3tGCKOxQevCN3y+2ft jffqMEqOCgrN4kvKivj75V3alFQotT+jbZm1nJEwuQCLSJCqiHWcyCLlJF9Y6axn Thvsv40bnTKCPgqezo/0AYiYjQ9xIatTC3QDw129E7bofNKPBLk7LWcbg9CQBu+T iboA8IIxFgrOFYn66Mgx4kcJcQTRJ2XgdJ1v8p+mSITWH3UkLa5OhZeTqU6x2LDl LPuY8eC6y9QUqpFeEtaL72ZpDdYAn7Vcu4B3+D4Oobh7o2EJNQijIQ6A2QKIFw6e eBACKL0JJMwvfxVnp3nKIuoB3yOemMGZ8NpqUNcEn5mjmZubRWXXJXjtjjF5pGYW RRbMXvs3tFhLGsqnjVHQ/AV5MyuYKfl4Tqhvrz0u2oh0A8uo5Kq3CuHBDcLhLjs1 RkDiZuSdVugRFeq6hcQAyqO6LQ/QRhqtQ1hscecr9Iv8grvs8gzi4PvlurgBFqEF AuWe0V2WY0IJ9S7BqUUDr3Ij+0CQgxeQ70yyOztWjsT0B6ZPdOChm5Meu1+qi2ni EuT6Q5Lo2KHTqhrvi/RbTUXs+D6LSNFN6QFOzWtKWAr+gyrQjKg= =Ew/J -----END PGP SIGNATURE----- --------------------------------------------------------------------- To unsubscribe, e-mail: users-unsubscr...@tomcat.apache.org For additional commands, e-mail: users-h...@tomcat.apache.org
