From: Christopher Schultz <ch...@christopherschultz.net>
Date: Wednesday, March 26, 2025 at 6:57 PM
To: users@tomcat.apache.org <users@tomcat.apache.org>
Subject: Re: NIO Thread Madness
William,
On 3/25/25 2:51 PM, William Crowell wrote:
Mark,
I think we might have found something. I think the DBCP2 connection pool is
returning stale connections from Oracle. There is no firewall between Tomcat
and Oracle, but I looked at the context.xml and found the following:
…
accessToUnderlyingConnectionAllowed=”true”
maxIdle=”100”
maxWaitMillis=”10000”
minIdle=”25”
validationQuery=”select 1 from dual”
url=”blah blah blah”
maxTotal=”true”
^ This is typically a number, not a true/false value.
logAbandoned=”true”
removeAbandonedOnBorrow=”true”
removeAbandonedTimeout=”900”
removeAbandonedOnMaintenance=”true”
timeBetweenEvictionRunsMillis=”300000” <!—5 minutes •
minEvictableIdleTimeMillis=”900000” <!—15 minutes •
The defaults for timeBetweenEvictionRunsMillis is 5 seconds and
minEvictableIdleTimeMillis is 60 seconds, and I also see removeAbandonedTimeout
is set to 15 minutes. Some of the queries to the database can run over 10
minutes. Sounds like an opportunity to recode this asynchronously.
Why this would cause Tomcat to go dark is beyond me. I will work on getting
new thread dumps and stack traces. Most of the long stack traces point to
issues with the database, and they are being sent over as screen shots. I’ll
see what I can do to work around that.
When the database pool dries up, your application will basically stop.
Setting removeAbandoned as you have is good, but with those long long
timeouts, they aren't doing any good.
I might set up a connection pool for "short queries" (like logging-in,
etc.) and then set up a separate pool for much longer queries.
Also, you might want to review your use of connections, etc. to ensure
that you are always closing everything in finally blocks. Resource
management with JDBC can be tedious and if you don't do it right, you
can leak connections from your pool very easily.
-chris
From: Mark Thomas <ma...@apache.org>
Date: Tuesday, March 25, 2025 at 1:13 PM
To: users@tomcat.apache.org <users@tomcat.apache.org>
Subject: Re: NIO Thread Madness
On 25/03/2025 12:33, William Crowell wrote:
Mark,
I believe there is a proxy involved here that does TLS decrypt, but I noticed
they had the redirectPort on the 8080 connector set to 8443. When you try to
hit Tomcat directly over port 8080 using HTTP it is hung.
Hmm. Both the Acceptor thread and the Poller thread are running and
appear to be in states that would enable new requests to be processed.
There isn't logging in those two components for normal operations. I
assume because of performance concerns.
Do you have any thread dumps from when you have clients attempting new
requests that are hanging? I'm wondering if processing is hanging
somewhere in Tomcat / the application during request processing.
I would probably be thinking about enabling remote debugging and
connecting a debugger the next time it goes wrong and tracing the
progress of a new request. But I accept that may not be practical for a
production system.
It seems unlikely that the new requests are hanging before they reach
the Tomcat code but that seems unlikely.
Other things to try:
netstat may shed some light on the current state of Tomcat and database
connections.
Wireshark or similar on the client side might also provide some clues.
I am wondering if Tomcat / Windows is responding at all, dropping the
connection, sending a reset, etc.
There might be some clues in the exceptions the exceptions that occur
just before the problem starts. Can you share any of them?
Mark
Regards,
William Crowell
From: Mark Thomas <ma...@apache.org>
Date: Tuesday, March 25, 2025 at 8:27 AM
To: users@tomcat.apache.org <users@tomcat.apache.org>
Subject: Re: NIO Thread Madness
On 25/03/2025 11:24, William Crowell wrote:
Chris,
Looking at JMX is the next step. I make a request and Tomcat never returns,
and I do not get a “connection refused”. It just sits and hangs.
Looking that the thread dump you sent me privately now.
Which port/protocol are you using to connect to Tomcat? HTTP and 8080?
Are you connecting directly to Tomcat or is there a proxy involved at all?
Mark
Regards,
William Crowell
From: Christopher Schultz <ch...@christopherschultz.net>
Date: Tuesday, March 25, 2025 at 7:20 AM
To: users@tomcat.apache.org <users@tomcat.apache.org>
Subject: Re: NIO Thread Madness
William,
On 3/24/25 2:56 PM, William Crowell wrote:
I am running Apache Tomcat 9.0.97 on Windows Server 2022. I’m running Oracle
JDK 1.8.0_371-b11 with a 4GB min heap and a 16GB max heap.
I have an application deployed on this server that is hitting an Oracle
database server. I have noticed the server stops accepting requests after
about 8-12 hours of uptime. In JProfiler you can tell when this is about to
happen because 20 of the 150 NIO threads BRIEFLY…BRIEFLY go into a blocked
state while querying the database. After this situation clears up, the NIO
thread pool grows slightly by about 15-20 threads, and then the application
server stops serving requests.
I looked at the GC log, and it looks completely healthy, and we are not even
close to our max heap. Metaspace size is not configured, but it looks fine
from the GC logs. There is no crash file or core dump produced.
I do notice some Oracle exceptions in the logs when this happens. We do have
about 1000 max connections defined on the Oracle database (which is too many).
Are you able to use JMX or similar to see how many used database
connections you have?
When you say that Tomcat does not accept requests, do you mean that you
make a request and it never returns, or do you mean that you get a
"connection refused" or something similar?
I have my thread pool defined as follows in server.xml:
…
<Executor name="tomcatThreadPool"
namePrefix="catalina-exec-"
maxThreads="1000"
minSpareThreads="50"
maxIdleTime="60000"
maxQueueSize="1000" />
<Connector port="8080"
protocol="org.apache.coyote.http11.Http11NioProtocol"
connectionTimeout="130000"
redirectPort="8443"
disableUploadTimeout="false"
acceptCount="1000"
maxConnections="1000"
executor="tomcatThreadPool" />
Have you tried enabling any of the "abandoned"-related settings? I
highly recommend logAbandoned="true" and possibly others.
I would imagine you'd see threads waiting on database connections if you
had exhausted your pool. Just remember that threads can be stuck on
things without being BLOCKED.
> Are there any logs I can enable to find out why the application
server stops accepting connections?
-chris
---------------------------------------------------------------------
To unsubscribe, e-mail: users-unsubscr...@tomcat.apache.org
For additional commands, e-mail: users-h...@tomcat.apache.org
CAUTION: This email originated from outside of the organization. Do not click
on links or open attachments unless you recognize the sender and know the
content is safe.
This e-mail may contain information that is privileged or confidential. If you
are not the intended recipient, please delete the e-mail and any attachments
and notify us immediately.
---------------------------------------------------------------------
To unsubscribe, e-mail: users-unsubscr...@tomcat.apache.org
For additional commands, e-mail: users-h...@tomcat.apache.org
CAUTION: This email originated from outside of the organization. Do not click
on links or open attachments unless you recognize the sender and know the
content is safe.
This e-mail may contain information that is privileged or confidential. If you
are not the intended recipient, please delete the e-mail and any attachments
and notify us immediately.
---------------------------------------------------------------------
To unsubscribe, e-mail: users-unsubscr...@tomcat.apache.org
For additional commands, e-mail: users-h...@tomcat.apache.org
CAUTION: This email originated from outside of the organization. Do not click
on links or open attachments unless you recognize the sender and know the
content is safe.
This e-mail may contain information that is privileged or confidential. If you
are not the intended recipient, please delete the e-mail and any attachments
and notify us immediately.
---------------------------------------------------------------------
To unsubscribe, e-mail: users-unsubscr...@tomcat.apache.org
For additional commands, e-mail: users-h...@tomcat.apache.org
CAUTION: This email originated from outside of the organization. Do not click
on links or open attachments unless you recognize the sender and know the
content is safe.
This e-mail may contain information that is privileged or confidential. If you
are not the intended recipient, please delete the e-mail and any attachments
and notify us immediately.
