Chris, Oracle’s ojdbc8.jar version 19.25.0.0.0 which should support JDBC 4.2:
https://www.oracle.com/database/technologies/maven-central-guide.html Regards, William Crowell From: Christopher Schultz <ch...@christopherschultz.net> Date: Monday, March 31, 2025 at 3:50 PM To: users@tomcat.apache.org <users@tomcat.apache.org> Subject: Re: NIO Thread Madness William, On 3/31/25 2:31 PM, William Crowell wrote: > Question related to this. I found issue DBCP-599 which was released > in DBCP 2.13.0 as part of Apache Tomcat 9.0.98 release. The > characteristics of this major bug appear very similar to the issue I am > having. Are you using an old driver that does not support JDBC 4.2? > In Tomcat’s lib directory there is a tomcat-dbcp.jar. Is that just > DBCP2 repackaged under Tomcat or is that an interface from Tomcat > into Apache Commons DBCP2? It's re-packaged. -chris > From: William Crowell <wcrow...@perforce.com.INVALID> > Date: Friday, March 28, 2025 at 12:22 PM > To: Tomcat Users List <users@tomcat.apache.org> > Subject: Re: NIO Thread Madness > Very good idea Chriis. Thank you. > > Regards, > > William Crowell > > From: Christopher Schultz <ch...@christopherschultz.net> > Date: Friday, March 28, 2025 at 12:05 PM > To: users@tomcat.apache.org <users@tomcat.apache.org> > Subject: Re: NIO Thread Madness > William, > > On 3/26/25 7:06 PM, William Crowell wrote: >> That maxTotal was a typo due to trying to copy the config from a >> screenshot. >> >> I agree with your assessment. I think there are 2 situations going >> on. One is that the code may not be properly closing connections, >> and the connection pool is not properly configured. > I wrote this a looong time ago, but it's very helpful to show to > programmers who haven't been careful. > > https://nam12.safelinks.protection.outlook.com/?url=https%3A%2F%2Fblog.christopherschultz.net%2F2009%2F03%2F16%2Fproperly-handling-pooled-jdbc-connections%2F&data=05%7C02%7CWCrowell%40perforce.com%7C5838853706b3439d53f708dd708d48a7%7C95b666d19a7549ab95a38969fbcdc08c%7C0%7C0%7C638790474303279804%7CUnknown%7CTWFpbGZsb3d8eyJFbXB0eU1hcGkiOnRydWUsIlYiOiIwLjAuMDAwMCIsIlAiOiJXaW4zMiIsIkFOIjoiTWFpbCIsIldUIjoyfQ%3D%3D%7C0%7C%7C%7C&sdata=n52OlWTJYXuUvy%2FPepZNxRtHjWOQbQlmo6zskRW28JU%3D&reserved=0<https://nam12.safelinks.protection.outlook.com/?url=https%3A%2F%2Fblog.christopherschultz.net%2F2009%2F03%2F16%2Fproperly-handling-pooled-jdbc-connections%2F&data=05%7C02%7CWCrowell%40perforce.com%7C5838853706b3439d53f708dd708d48a7%7C95b666d19a7549ab95a38969fbcdc08c%7C0%7C0%7C638790474303299823%7CUnknown%7CTWFpbGZsb3d8eyJFbXB0eU1hcGkiOnRydWUsIlYiOiIwLjAuMDAwMCIsIlAiOiJXaW4zMiIsIkFOIjoiTWFpbCIsIldUIjoyfQ%3D%3D%7C0%7C%7C%7C&sdata=fIYe4E6WYeBXonauF%2B3mUeQs9EnIG5NeB8ovsiXZais%3D&reserved=0><https://nam12.safelinks.protection.outlook.com/?url=https%3A%2F%2Fblog.christopherschultz.net%2F2009%2F03%2F16%2Fproperly-handling-pooled-jdbc-connections%2F&data=05%7C02%7CWCrowell%40perforce.com%7C5838853706b3439d53f708dd708d48a7%7C95b666d19a7549ab95a38969fbcdc08c%7C0%7C0%7C638790474303310799%7CUnknown%7CTWFpbGZsb3d8eyJFbXB0eU1hcGkiOnRydWUsIlYiOiIwLjAuMDAwMCIsIlAiOiJXaW4zMiIsIkFOIjoiTWFpbCIsIldUIjoyfQ%3D%3D%7C0%7C%7C%7C&sdata=h6TJBfYtU3ZA4MY5cyPE%2BOl4KitCpIPCUhQo9P0VJDM%3D&reserved=0><https://blog.christopherschultz.net/2009/03/16/properly-handling-pooled-jdbc-connections/> > > For your long-transactions, I think you are going to want to use a > separate pool. > > -chris > >> From: Christopher Schultz <ch...@christopherschultz.net> >> Date: Wednesday, March 26, 2025 at 6:57 PM >> To: users@tomcat.apache.org <users@tomcat.apache.org> >> Subject: Re: NIO Thread Madness >> William, >> >> On 3/25/25 2:51 PM, William Crowell wrote: >>> Mark, >>> >>> I think we might have found something. I think the DBCP2 connection pool >>> is returning stale connections from Oracle. There is no firewall between >>> Tomcat and Oracle, but I looked at the context.xml and found the following: >>> >>> … >>> accessToUnderlyingConnectionAllowed=”true” >>> maxIdle=”100” >>> maxWaitMillis=”10000” >>> minIdle=”25” >>> validationQuery=”select 1 from dual” >>> url=”blah blah blah” >>> maxTotal=”true” >> >> ^ This is typically a number, not a true/false value. >> >>> logAbandoned=”true” >>> removeAbandonedOnBorrow=”true” >>> removeAbandonedTimeout=”900” >>> removeAbandonedOnMaintenance=”true” >>> timeBetweenEvictionRunsMillis=”300000” <!—5 minutes • >>> minEvictableIdleTimeMillis=”900000” <!—15 minutes • >>> >>> The defaults for timeBetweenEvictionRunsMillis is 5 seconds and >>> minEvictableIdleTimeMillis is 60 seconds, and I also see >>> removeAbandonedTimeout is set to 15 minutes. Some of the queries to the >>> database can run over 10 minutes. Sounds like an opportunity to recode >>> this asynchronously. >>> >>> Why this would cause Tomcat to go dark is beyond me. I will work on >>> getting new thread dumps and stack traces. Most of the long stack traces >>> point to issues with the database, and they are being sent over as screen >>> shots. I’ll see what I can do to work around that. >> >> When the database pool dries up, your application will basically stop. >> Setting removeAbandoned as you have is good, but with those long long >> timeouts, they aren't doing any good. >> >> I might set up a connection pool for "short queries" (like logging-in, >> etc.) and then set up a separate pool for much longer queries. >> >> Also, you might want to review your use of connections, etc. to ensure >> that you are always closing everything in finally blocks. Resource >> management with JDBC can be tedious and if you don't do it right, you >> can leak connections from your pool very easily. >> >> -chris >> >>> From: Mark Thomas <ma...@apache.org> >>> Date: Tuesday, March 25, 2025 at 1:13 PM >>> To: users@tomcat.apache.org <users@tomcat.apache.org> >>> Subject: Re: NIO Thread Madness >>> On 25/03/2025 12:33, William Crowell wrote: >>>> Mark, >>>> >>>> I believe there is a proxy involved here that does TLS decrypt, but I >>>> noticed they had the redirectPort on the 8080 connector set to 8443. When >>>> you try to hit Tomcat directly over port 8080 using HTTP it is hung. >>> >>> Hmm. Both the Acceptor thread and the Poller thread are running and >>> appear to be in states that would enable new requests to be processed. >>> >>> There isn't logging in those two components for normal operations. I >>> assume because of performance concerns. >>> >>> Do you have any thread dumps from when you have clients attempting new >>> requests that are hanging? I'm wondering if processing is hanging >>> somewhere in Tomcat / the application during request processing. >>> >>> I would probably be thinking about enabling remote debugging and >>> connecting a debugger the next time it goes wrong and tracing the >>> progress of a new request. But I accept that may not be practical for a >>> production system. >>> >>> It seems unlikely that the new requests are hanging before they reach >>> the Tomcat code but that seems unlikely. >>> >>> Other things to try: >>> >>> netstat may shed some light on the current state of Tomcat and database >>> connections. >>> >>> Wireshark or similar on the client side might also provide some clues. >>> >>> I am wondering if Tomcat / Windows is responding at all, dropping the >>> connection, sending a reset, etc. >>> >>> There might be some clues in the exceptions the exceptions that occur >>> just before the problem starts. Can you share any of them? >>> >>> Mark >>> >>>> >>>> Regards, >>>> >>>> William Crowell >>>> >>>> From: Mark Thomas <ma...@apache.org> >>>> Date: Tuesday, March 25, 2025 at 8:27 AM >>>> To: users@tomcat.apache.org <users@tomcat.apache.org> >>>> Subject: Re: NIO Thread Madness >>>> On 25/03/2025 11:24, William Crowell wrote: >>>>> Chris, >>>>> >>>>> Looking at JMX is the next step. I make a request and Tomcat never >>>>> returns, and I do not get a “connection refused”. It just sits and hangs. >>>> >>>> Looking that the thread dump you sent me privately now. >>>> >>>> Which port/protocol are you using to connect to Tomcat? HTTP and 8080? >>>> >>>> Are you connecting directly to Tomcat or is there a proxy involved at all? >>>> >>>> Mark >>>> >>>>> >>>>> Regards, >>>>> >>>>> William Crowell >>>>> >>>>> From: Christopher Schultz <ch...@christopherschultz.net> >>>>> Date: Tuesday, March 25, 2025 at 7:20 AM >>>>> To: users@tomcat.apache.org <users@tomcat.apache.org> >>>>> Subject: Re: NIO Thread Madness >>>>> William, >>>>> >>>>> On 3/24/25 2:56 PM, William Crowell wrote: >>>>>> I am running Apache Tomcat 9.0.97 on Windows Server 2022. I’m running >>>>>> Oracle JDK 1.8.0_371-b11 with a 4GB min heap and a 16GB max heap. >>>>>> >>>>>> I have an application deployed on this server that is hitting an Oracle >>>>>> database server. I have noticed the server stops accepting requests >>>>>> after about 8-12 hours of uptime. In JProfiler you can tell when this >>>>>> is about to happen because 20 of the 150 NIO threads BRIEFLY…BRIEFLY go >>>>>> into a blocked state while querying the database. After this situation >>>>>> clears up, the NIO thread pool grows slightly by about 15-20 threads, >>>>>> and then the application server stops serving requests. >>>>>> >>>>>> I looked at the GC log, and it looks completely healthy, and we are not >>>>>> even close to our max heap. Metaspace size is not configured, but it >>>>>> looks fine from the GC logs. There is no crash file or core dump >>>>>> produced. >>>>>> >>>>>> I do notice some Oracle exceptions in the logs when this happens. We do >>>>>> have about 1000 max connections defined on the Oracle database (which is >>>>>> too many). >>>>> >>>>> Are you able to use JMX or similar to see how many used database >>>>> connections you have? >>>>> >>>>> When you say that Tomcat does not accept requests, do you mean that you >>>>> make a request and it never returns, or do you mean that you get a >>>>> "connection refused" or something similar? >>>>> >>>>>> I have my thread pool defined as follows in server.xml: >>>>>> >>>>>> … >>>>>> <Executor name="tomcatThreadPool" >>>>>> namePrefix="catalina-exec-" >>>>>> maxThreads="1000" >>>>>> minSpareThreads="50" >>>>>> maxIdleTime="60000" >>>>>> maxQueueSize="1000" /> >>>>>> >>>>>> <Connector port="8080" >>>>>> protocol="org.apache.coyote.http11.Http11NioProtocol" >>>>>> connectionTimeout="130000" >>>>>> redirectPort="8443" >>>>>> disableUploadTimeout="false" >>>>>> acceptCount="1000" >>>>>> maxConnections="1000" >>>>>> executor="tomcatThreadPool" /> >>>>> >>>>> Have you tried enabling any of the "abandoned"-related settings? I >>>>> highly recommend logAbandoned="true" and possibly others. >>>>> >>>>> I would imagine you'd see threads waiting on database connections if you >>>>> had exhausted your pool. Just remember that threads can be stuck on >>>>> things without being BLOCKED. >>>>> >>>>> > Are there any logs I can enable to find out why the application >>>>> server stops accepting connections? >>>>> >>>>> -chris >>>>> >>>>> >>>>> --------------------------------------------------------------------- >>>>> To unsubscribe, e-mail: users-unsubscr...@tomcat.apache.org >>>>> For additional commands, e-mail: users-h...@tomcat.apache.org >>>>> >>>>> >>>>> >>>>> CAUTION: This email originated from outside of the organization. Do not >>>>> click on links or open attachments unless you recognize the sender and >>>>> know the content is safe. >>>>> >>>>> >>>>> This e-mail may contain information that is privileged or confidential. >>>>> If you are not the intended recipient, please delete the e-mail and any >>>>> attachments and notify us immediately. >>>>> >>>> >>>> >>>> --------------------------------------------------------------------- >>>> To unsubscribe, e-mail: users-unsubscr...@tomcat.apache.org >>>> For additional commands, e-mail: users-h...@tomcat.apache.org >>>> >>>> >>>> >>>> CAUTION: This email originated from outside of the organization. Do not >>>> click on links or open attachments unless you recognize the sender and >>>> know the content is safe. >>>> >>>> >>>> This e-mail may contain information that is privileged or confidential. If >>>> you are not the intended recipient, please delete the e-mail and any >>>> attachments and notify us immediately. >>>> >>> >>> >>> --------------------------------------------------------------------- >>> To unsubscribe, e-mail: users-unsubscr...@tomcat.apache.org >>> For additional commands, e-mail: users-h...@tomcat.apache.org >>> >>> >>> >>> CAUTION: This email originated from outside of the organization. Do not >>> click on links or open attachments unless you recognize the sender and know >>> the content is safe. >>> >>> >>> This e-mail may contain information that is privileged or confidential. If >>> you are not the intended recipient, please delete the e-mail and any >>> attachments and notify us immediately. >>> >> >> >> --------------------------------------------------------------------- >> To unsubscribe, e-mail: users-unsubscr...@tomcat.apache.org >> For additional commands, e-mail: users-h...@tomcat.apache.org >> >> >> >> CAUTION: This email originated from outside of the organization. Do not >> click on links or open attachments unless you recognize the sender and know >> the content is safe. >> >> >> This e-mail may contain information that is privileged or confidential. If >> you are not the intended recipient, please delete the e-mail and any >> attachments and notify us immediately. >> > > > --------------------------------------------------------------------- > To unsubscribe, e-mail: users-unsubscr...@tomcat.apache.org > For additional commands, e-mail: users-h...@tomcat.apache.org > > > > CAUTION: This email originated from outside of the organization. Do not click > on links or open attachments unless you recognize the sender and know the > content is safe. > > > This e-mail may contain information that is privileged or confidential. If > you are not the intended recipient, please delete the e-mail and any > attachments and notify us immediately. > > > This e-mail may contain information that is privileged or confidential. If > you are not the intended recipient, please delete the e-mail and any > attachments and notify us immediately. > Т���������������������������������������������������������������������ХF�V�7V'67&�&R�R���âW6W'2�V�7V'67&�&TF��6B�6�R��&pФf�"FF�F����6����G2�R���âW6W'2ֆV�F��6B�6�R��&pР CAUTION: This email originated from outside of the organization. Do not click on links or open attachments unless you recognize the sender and know the content is safe. This e-mail may contain information that is privileged or confidential. If you are not the intended recipient, please delete the e-mail and any attachments and notify us immediately.
