Very good idea Chriis. Thank you. Regards,
William Crowell From: Christopher Schultz <ch...@christopherschultz.net> Date: Friday, March 28, 2025 at 12:05 PM To: users@tomcat.apache.org <users@tomcat.apache.org> Subject: Re: NIO Thread Madness William, On 3/26/25 7:06 PM, William Crowell wrote: > That maxTotal was a typo due to trying to copy the config from a > screenshot. > > I agree with your assessment. I think there are 2 situations going > on. One is that the code may not be properly closing connections, > and the connection pool is not properly configured. I wrote this a looong time ago, but it's very helpful to show to programmers who haven't been careful. https://nam12.safelinks.protection.outlook.com/?url=https%3A%2F%2Fblog.christopherschultz.net%2F2009%2F03%2F16%2Fproperly-handling-pooled-jdbc-connections%2F&data=05%7C02%7CWCrowell%40perforce.com%7C94d759d31b034232774108dd6e1255ee%7C95b666d19a7549ab95a38969fbcdc08c%7C0%7C0%7C638787747219563730%7CUnknown%7CTWFpbGZsb3d8eyJFbXB0eU1hcGkiOnRydWUsIlYiOiIwLjAuMDAwMCIsIlAiOiJXaW4zMiIsIkFOIjoiTWFpbCIsIldUIjoyfQ%3D%3D%7C0%7C%7C%7C&sdata=IjI0vAoxrQ53I%2FTByJRoQ4mRVYmG746pnENm8JUlq6Q%3D&reserved=0<https://blog.christopherschultz.net/2009/03/16/properly-handling-pooled-jdbc-connections/> For your long-transactions, I think you are going to want to use a separate pool. -chris > From: Christopher Schultz <ch...@christopherschultz.net> > Date: Wednesday, March 26, 2025 at 6:57 PM > To: users@tomcat.apache.org <users@tomcat.apache.org> > Subject: Re: NIO Thread Madness > William, > > On 3/25/25 2:51 PM, William Crowell wrote: >> Mark, >> >> I think we might have found something. I think the DBCP2 connection pool is >> returning stale connections from Oracle. There is no firewall between >> Tomcat and Oracle, but I looked at the context.xml and found the following: >> >> … >> accessToUnderlyingConnectionAllowed=”true” >> maxIdle=”100” >> maxWaitMillis=”10000” >> minIdle=”25” >> validationQuery=”select 1 from dual” >> url=”blah blah blah” >> maxTotal=”true” > > ^ This is typically a number, not a true/false value. > >> logAbandoned=”true” >> removeAbandonedOnBorrow=”true” >> removeAbandonedTimeout=”900” >> removeAbandonedOnMaintenance=”true” >> timeBetweenEvictionRunsMillis=”300000” <!—5 minutes • >> minEvictableIdleTimeMillis=”900000” <!—15 minutes • >> >> The defaults for timeBetweenEvictionRunsMillis is 5 seconds and >> minEvictableIdleTimeMillis is 60 seconds, and I also see >> removeAbandonedTimeout is set to 15 minutes. Some of the queries to the >> database can run over 10 minutes. Sounds like an opportunity to recode this >> asynchronously. >> >> Why this would cause Tomcat to go dark is beyond me. I will work on getting >> new thread dumps and stack traces. Most of the long stack traces point to >> issues with the database, and they are being sent over as screen shots. >> I’ll see what I can do to work around that. > > When the database pool dries up, your application will basically stop. > Setting removeAbandoned as you have is good, but with those long long > timeouts, they aren't doing any good. > > I might set up a connection pool for "short queries" (like logging-in, > etc.) and then set up a separate pool for much longer queries. > > Also, you might want to review your use of connections, etc. to ensure > that you are always closing everything in finally blocks. Resource > management with JDBC can be tedious and if you don't do it right, you > can leak connections from your pool very easily. > > -chris > >> From: Mark Thomas <ma...@apache.org> >> Date: Tuesday, March 25, 2025 at 1:13 PM >> To: users@tomcat.apache.org <users@tomcat.apache.org> >> Subject: Re: NIO Thread Madness >> On 25/03/2025 12:33, William Crowell wrote: >>> Mark, >>> >>> I believe there is a proxy involved here that does TLS decrypt, but I >>> noticed they had the redirectPort on the 8080 connector set to 8443. When >>> you try to hit Tomcat directly over port 8080 using HTTP it is hung. >> >> Hmm. Both the Acceptor thread and the Poller thread are running and >> appear to be in states that would enable new requests to be processed. >> >> There isn't logging in those two components for normal operations. I >> assume because of performance concerns. >> >> Do you have any thread dumps from when you have clients attempting new >> requests that are hanging? I'm wondering if processing is hanging >> somewhere in Tomcat / the application during request processing. >> >> I would probably be thinking about enabling remote debugging and >> connecting a debugger the next time it goes wrong and tracing the >> progress of a new request. But I accept that may not be practical for a >> production system. >> >> It seems unlikely that the new requests are hanging before they reach >> the Tomcat code but that seems unlikely. >> >> Other things to try: >> >> netstat may shed some light on the current state of Tomcat and database >> connections. >> >> Wireshark or similar on the client side might also provide some clues. >> >> I am wondering if Tomcat / Windows is responding at all, dropping the >> connection, sending a reset, etc. >> >> There might be some clues in the exceptions the exceptions that occur >> just before the problem starts. Can you share any of them? >> >> Mark >> >>> >>> Regards, >>> >>> William Crowell >>> >>> From: Mark Thomas <ma...@apache.org> >>> Date: Tuesday, March 25, 2025 at 8:27 AM >>> To: users@tomcat.apache.org <users@tomcat.apache.org> >>> Subject: Re: NIO Thread Madness >>> On 25/03/2025 11:24, William Crowell wrote: >>>> Chris, >>>> >>>> Looking at JMX is the next step. I make a request and Tomcat never >>>> returns, and I do not get a “connection refused”. It just sits and hangs. >>> >>> Looking that the thread dump you sent me privately now. >>> >>> Which port/protocol are you using to connect to Tomcat? HTTP and 8080? >>> >>> Are you connecting directly to Tomcat or is there a proxy involved at all? >>> >>> Mark >>> >>>> >>>> Regards, >>>> >>>> William Crowell >>>> >>>> From: Christopher Schultz <ch...@christopherschultz.net> >>>> Date: Tuesday, March 25, 2025 at 7:20 AM >>>> To: users@tomcat.apache.org <users@tomcat.apache.org> >>>> Subject: Re: NIO Thread Madness >>>> William, >>>> >>>> On 3/24/25 2:56 PM, William Crowell wrote: >>>>> I am running Apache Tomcat 9.0.97 on Windows Server 2022. I’m running >>>>> Oracle JDK 1.8.0_371-b11 with a 4GB min heap and a 16GB max heap. >>>>> >>>>> I have an application deployed on this server that is hitting an Oracle >>>>> database server. I have noticed the server stops accepting requests >>>>> after about 8-12 hours of uptime. In JProfiler you can tell when this is >>>>> about to happen because 20 of the 150 NIO threads BRIEFLY…BRIEFLY go into >>>>> a blocked state while querying the database. After this situation clears >>>>> up, the NIO thread pool grows slightly by about 15-20 threads, and then >>>>> the application server stops serving requests. >>>>> >>>>> I looked at the GC log, and it looks completely healthy, and we are not >>>>> even close to our max heap. Metaspace size is not configured, but it >>>>> looks fine from the GC logs. There is no crash file or core dump >>>>> produced. >>>>> >>>>> I do notice some Oracle exceptions in the logs when this happens. We do >>>>> have about 1000 max connections defined on the Oracle database (which is >>>>> too many). >>>> >>>> Are you able to use JMX or similar to see how many used database >>>> connections you have? >>>> >>>> When you say that Tomcat does not accept requests, do you mean that you >>>> make a request and it never returns, or do you mean that you get a >>>> "connection refused" or something similar? >>>> >>>>> I have my thread pool defined as follows in server.xml: >>>>> >>>>> … >>>>> <Executor name="tomcatThreadPool" >>>>> namePrefix="catalina-exec-" >>>>> maxThreads="1000" >>>>> minSpareThreads="50" >>>>> maxIdleTime="60000" >>>>> maxQueueSize="1000" /> >>>>> >>>>> <Connector port="8080" >>>>> protocol="org.apache.coyote.http11.Http11NioProtocol" >>>>> connectionTimeout="130000" >>>>> redirectPort="8443" >>>>> disableUploadTimeout="false" >>>>> acceptCount="1000" >>>>> maxConnections="1000" >>>>> executor="tomcatThreadPool" /> >>>> >>>> Have you tried enabling any of the "abandoned"-related settings? I >>>> highly recommend logAbandoned="true" and possibly others. >>>> >>>> I would imagine you'd see threads waiting on database connections if you >>>> had exhausted your pool. Just remember that threads can be stuck on >>>> things without being BLOCKED. >>>> >>>> > Are there any logs I can enable to find out why the application >>>> server stops accepting connections? >>>> >>>> -chris >>>> >>>> >>>> --------------------------------------------------------------------- >>>> To unsubscribe, e-mail: users-unsubscr...@tomcat.apache.org >>>> For additional commands, e-mail: users-h...@tomcat.apache.org >>>> >>>> >>>> >>>> CAUTION: This email originated from outside of the organization. Do not >>>> click on links or open attachments unless you recognize the sender and >>>> know the content is safe. >>>> >>>> >>>> This e-mail may contain information that is privileged or confidential. If >>>> you are not the intended recipient, please delete the e-mail and any >>>> attachments and notify us immediately. >>>> >>> >>> >>> --------------------------------------------------------------------- >>> To unsubscribe, e-mail: users-unsubscr...@tomcat.apache.org >>> For additional commands, e-mail: users-h...@tomcat.apache.org >>> >>> >>> >>> CAUTION: This email originated from outside of the organization. Do not >>> click on links or open attachments unless you recognize the sender and know >>> the content is safe. >>> >>> >>> This e-mail may contain information that is privileged or confidential. If >>> you are not the intended recipient, please delete the e-mail and any >>> attachments and notify us immediately. >>> >> >> >> --------------------------------------------------------------------- >> To unsubscribe, e-mail: users-unsubscr...@tomcat.apache.org >> For additional commands, e-mail: users-h...@tomcat.apache.org >> >> >> >> CAUTION: This email originated from outside of the organization. Do not >> click on links or open attachments unless you recognize the sender and know >> the content is safe. >> >> >> This e-mail may contain information that is privileged or confidential. If >> you are not the intended recipient, please delete the e-mail and any >> attachments and notify us immediately. >> > > > --------------------------------------------------------------------- > To unsubscribe, e-mail: users-unsubscr...@tomcat.apache.org > For additional commands, e-mail: users-h...@tomcat.apache.org > > > > CAUTION: This email originated from outside of the organization. Do not click > on links or open attachments unless you recognize the sender and know the > content is safe. > > > This e-mail may contain information that is privileged or confidential. If > you are not the intended recipient, please delete the e-mail and any > attachments and notify us immediately. > --------------------------------------------------------------------- To unsubscribe, e-mail: users-unsubscr...@tomcat.apache.org For additional commands, e-mail: users-h...@tomcat.apache.org CAUTION: This email originated from outside of the organization. Do not click on links or open attachments unless you recognize the sender and know the content is safe. This e-mail may contain information that is privileged or confidential. If you are not the intended recipient, please delete the e-mail and any attachments and notify us immediately.
