Mark, I think we might have found something. I think the DBCP2 connection pool is returning stale connections from Oracle. There is no firewall between Tomcat and Oracle, but I looked at the context.xml and found the following:
… accessToUnderlyingConnectionAllowed=”true” maxIdle=”100” maxWaitMillis=”10000” minIdle=”25” validationQuery=”select 1 from dual” url=”blah blah blah” maxTotal=”true” logAbandoned=”true” removeAbandonedOnBorrow=”true” removeAbandonedTimeout=”900” removeAbandonedOnMaintenance=”true” timeBetweenEvictionRunsMillis=”300000” <!—5 minutes • minEvictableIdleTimeMillis=”900000” <!—15 minutes • The defaults for timeBetweenEvictionRunsMillis is 5 seconds and minEvictableIdleTimeMillis is 60 seconds, and I also see removeAbandonedTimeout is set to 15 minutes. Some of the queries to the database can run over 10 minutes. Sounds like an opportunity to recode this asynchronously. Why this would cause Tomcat to go dark is beyond me. I will work on getting new thread dumps and stack traces. Most of the long stack traces point to issues with the database, and they are being sent over as screen shots. I’ll see what I can do to work around that. Regards, William Crowell From: Mark Thomas <ma...@apache.org> Date: Tuesday, March 25, 2025 at 1:13 PM To: users@tomcat.apache.org <users@tomcat.apache.org> Subject: Re: NIO Thread Madness On 25/03/2025 12:33, William Crowell wrote: > Mark, > > I believe there is a proxy involved here that does TLS decrypt, but I noticed > they had the redirectPort on the 8080 connector set to 8443. When you try to > hit Tomcat directly over port 8080 using HTTP it is hung. Hmm. Both the Acceptor thread and the Poller thread are running and appear to be in states that would enable new requests to be processed. There isn't logging in those two components for normal operations. I assume because of performance concerns. Do you have any thread dumps from when you have clients attempting new requests that are hanging? I'm wondering if processing is hanging somewhere in Tomcat / the application during request processing. I would probably be thinking about enabling remote debugging and connecting a debugger the next time it goes wrong and tracing the progress of a new request. But I accept that may not be practical for a production system. It seems unlikely that the new requests are hanging before they reach the Tomcat code but that seems unlikely. Other things to try: netstat may shed some light on the current state of Tomcat and database connections. Wireshark or similar on the client side might also provide some clues. I am wondering if Tomcat / Windows is responding at all, dropping the connection, sending a reset, etc. There might be some clues in the exceptions the exceptions that occur just before the problem starts. Can you share any of them? Mark > > Regards, > > William Crowell > > From: Mark Thomas <ma...@apache.org> > Date: Tuesday, March 25, 2025 at 8:27 AM > To: users@tomcat.apache.org <users@tomcat.apache.org> > Subject: Re: NIO Thread Madness > On 25/03/2025 11:24, William Crowell wrote: >> Chris, >> >> Looking at JMX is the next step. I make a request and Tomcat never returns, >> and I do not get a “connection refused”. It just sits and hangs. > > Looking that the thread dump you sent me privately now. > > Which port/protocol are you using to connect to Tomcat? HTTP and 8080? > > Are you connecting directly to Tomcat or is there a proxy involved at all? > > Mark > >> >> Regards, >> >> William Crowell >> >> From: Christopher Schultz <ch...@christopherschultz.net> >> Date: Tuesday, March 25, 2025 at 7:20 AM >> To: users@tomcat.apache.org <users@tomcat.apache.org> >> Subject: Re: NIO Thread Madness >> William, >> >> On 3/24/25 2:56 PM, William Crowell wrote: >>> I am running Apache Tomcat 9.0.97 on Windows Server 2022. I’m running >>> Oracle JDK 1.8.0_371-b11 with a 4GB min heap and a 16GB max heap. >>> >>> I have an application deployed on this server that is hitting an Oracle >>> database server. I have noticed the server stops accepting requests after >>> about 8-12 hours of uptime. In JProfiler you can tell when this is about >>> to happen because 20 of the 150 NIO threads BRIEFLY…BRIEFLY go into a >>> blocked state while querying the database. After this situation clears up, >>> the NIO thread pool grows slightly by about 15-20 threads, and then the >>> application server stops serving requests. >>> >>> I looked at the GC log, and it looks completely healthy, and we are not >>> even close to our max heap. Metaspace size is not configured, but it looks >>> fine from the GC logs. There is no crash file or core dump produced. >>> >>> I do notice some Oracle exceptions in the logs when this happens. We do >>> have about 1000 max connections defined on the Oracle database (which is >>> too many). >> >> Are you able to use JMX or similar to see how many used database >> connections you have? >> >> When you say that Tomcat does not accept requests, do you mean that you >> make a request and it never returns, or do you mean that you get a >> "connection refused" or something similar? >> >>> I have my thread pool defined as follows in server.xml: >>> >>> … >>> <Executor name="tomcatThreadPool" >>> namePrefix="catalina-exec-" >>> maxThreads="1000" >>> minSpareThreads="50" >>> maxIdleTime="60000" >>> maxQueueSize="1000" /> >>> >>> <Connector port="8080" >>> protocol="org.apache.coyote.http11.Http11NioProtocol" >>> connectionTimeout="130000" >>> redirectPort="8443" >>> disableUploadTimeout="false" >>> acceptCount="1000" >>> maxConnections="1000" >>> executor="tomcatThreadPool" /> >> >> Have you tried enabling any of the "abandoned"-related settings? I >> highly recommend logAbandoned="true" and possibly others. >> >> I would imagine you'd see threads waiting on database connections if you >> had exhausted your pool. Just remember that threads can be stuck on >> things without being BLOCKED. >> >> > Are there any logs I can enable to find out why the application >> server stops accepting connections? >> >> -chris >> >> >> --------------------------------------------------------------------- >> To unsubscribe, e-mail: users-unsubscr...@tomcat.apache.org >> For additional commands, e-mail: users-h...@tomcat.apache.org >> >> >> >> CAUTION: This email originated from outside of the organization. Do not >> click on links or open attachments unless you recognize the sender and know >> the content is safe. >> >> >> This e-mail may contain information that is privileged or confidential. If >> you are not the intended recipient, please delete the e-mail and any >> attachments and notify us immediately. >> > > > --------------------------------------------------------------------- > To unsubscribe, e-mail: users-unsubscr...@tomcat.apache.org > For additional commands, e-mail: users-h...@tomcat.apache.org > > > > CAUTION: This email originated from outside of the organization. Do not click > on links or open attachments unless you recognize the sender and know the > content is safe. > > > This e-mail may contain information that is privileged or confidential. If > you are not the intended recipient, please delete the e-mail and any > attachments and notify us immediately. > --------------------------------------------------------------------- To unsubscribe, e-mail: users-unsubscr...@tomcat.apache.org For additional commands, e-mail: users-h...@tomcat.apache.org CAUTION: This email originated from outside of the organization. Do not click on links or open attachments unless you recognize the sender and know the content is safe. This e-mail may contain information that is privileged or confidential. If you are not the intended recipient, please delete the e-mail and any attachments and notify us immediately.
