John,
On 9/30/24 10:53, John Williams wrote:
Sorry - didn't realize that images would be stripped off.
The number of TIMED_WAITING threads starts at 150 or so and every hour
increases by 10.
Below is an example of where the threads are stuck (by looking at the JVM
stack):
https-jsse-nio2-443-exec-90
TIMED_WAITING on
java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@48986fd7
jdk.internal.misc.Unsafe.park(Native Method);
java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:252);
java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:1674);
java.util.concurrent.LinkedBlockingQueue.poll(LinkedBlockingQueue.java:460);
app//org.apache.tomcat.util.threads.TaskQueue.poll(TaskQueue.java:99);
app//org.apache.tomcat.util.threads.TaskQueue.poll(TaskQueue.java:33);
app//org.apache.tomcat.util.threads.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1113);
app//org.apache.tomcat.util.threads.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1175);
app//org.apache.tomcat.util.threads.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:659);
app//org.apache.tomcat.util.threads.TaskThread$WrappingRunnable.run(TaskThread.java:63);
java.base@17.0.12/java.lang.Thread.run(Thread.java:840);
The connector configuration is:
<Connector protocol="org.apache.coyote.http11.Http11Nio2Protocol"
port="443" maxThreads="1536" minSpareThreads="64"
sslImplementationName="org.apache.tomcat.util.net.jsse.JSSEImplementation"
scheme="https" secure="true" SSLEnabled="true"
enableLookups="false" acceptCount="12000" connectionTimeout="60000" socket.rxBufSize="131072" socket.txBufSize="131072" maxHttpHeaderSize="131072"
maxPostSize="-1" maxKeepAliveRequests="1" maxConnections="1536" URIEncoding="UTF-8" useSendfile="false" tcpNoDelay="true" compression="force"
compressibleMimeType="font/woff,font/woff2,text/html,text/xml,text/plain,application/x-java-applet,application/octet-stream,application/xml,text/javascript,text/css,image/png,image/bmp,image/jpeg,image/gif,application/pdf,application/x-javascript,application/javascript,application/json,application/x-shockwave-flash,application/xhtml+xml,application/xml+xhtml"
noCompressionUserAgents="gozilla, traviata">
<SSLHostConfig honorCipherOrder="true"
ciphers="TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384,TLS_ECDHE_ECDSA_WITH_AES_256_GCM_SHA384,TLS_ECDH_ECDSA_WITH_AES_256_GCM_SHA384,TLS_ECDH_RSA_WITH_AES_256_GCM_SHA384,TLS_ECDHE_RSA_WITH_AES_128_GCM_SHA256,TLS_ECDHE_RSA_WITH_AES_256_CBC_SHA384,TLS_ECDHE_RSA_WITH_AES_128_CBC_SHA256,TLS_ECDHE_RSA_WITH_AES_256_CBC_SHA,TLS_ECDHE_RSA_WITH_AES_128_CBC_SHA,TLS_ECDHE_RSA_WITH_AES_256_CBC_SHA"
certificateVerification="none" protocols="TLSv1.2" sslProtocol="TLS"><Certificate certificateKeystoreFile="..."
certificateKeyAlias="..." certificateKeystorePassword="..." /></SSLHostConfig>
</Connector>
You are not explicitly using an <Executor>. When you configure using
<Connector> only, Tomcat will never retire the threads from the
thread-pool. This is mostly historic and while it could be improved, it
is better to switch to a mode modern configuration.
If you use <Connector executor="..."> and:
<Executor name="..."
maxThreads="1500"
minSpareThreads="64" />
Then you should see the number of threads fluxuate based upon the
request volume.
I'm not sure why you were seeing different behavior with Tomcat 10.0.
IIRC this behavior has been consistent as long as <Connector> and
<Executor> have been separate configuration elements.
-chris
-----Original Message-----
From: Christopher Schultz <ch...@christopherschultz.net>
Sent: Monday, September 30, 2024 10:38 AM
To: users@tomcat.apache.org
Subject: Re: Tomcat 10.1.30 and Many Timed Waiting threads
[You don't often get email from ch...@christopherschultz.net. Learn why this is
important at https://aka.ms/LearnAboutSenderIdentification ]
CAUTION: This email originated from outside of the organization. Do not click
links or open attachments unless you recognize the sender and know the content
is safe.
John,
On 9/30/24 10:30, John Williams wrote:
The issue is that the number of threads in the TIMED_WAIT state keeps
increasing over time. Starts at 150 and keeps growing by 10 every hour.
Once it reaches close to the maxThreads setting we see connection drops.
This application was working fine with Tomcat 10.0 and JDK 12. Unless
the application was seeing load, threads would not be created in
Tomcat's JVM.
You can also see the thread stacks in this image.
This list strips out image. Can you post text?
Please post your <Connector> configuration taking care to remove any secrets.
Thanks,
-chris
-----Original Message-----
From: Christopher Schultz <ch...@christopherschultz.net>
Sent: Monday, September 30, 2024 10:20 AM
To: users@tomcat.apache.org
Subject: Re: Tomcat 10.1.30 and Many Timed Waiting threads
[You don't often get email from ch...@christopherschultz.net
<mailto:ch...@christopherschultz.net>. Learn why this is important at
https://aka.ms/LearnAboutSenderIdentification
<https://apc01.safelinks.protection.outlook.com/?url=https%3A%2F%2Faka
.ms%2F&data=05%7C02%7Csrinivas%40eginnovations.com%7C5c93ec3de0884e786
7d708dce15d7e73%7C62c210cb17214d259b6b0c95ceb472ce%7C0%7C0%7C638633038
912339442%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIi
LCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C0%7C%7C%7C&sdata=6q8HQHLjceSHYLqEnLN9
ue%2Fpa3zz5cougsajpZIsjDw%3D&reserved=0
LearnAboutSenderIdentification> ]
CAUTION: This email originated from outside of the organization. Do
not click links or open attachments unless you recognize the sender
and know the content is safe.
John,
On 9/28/24 05:38, John Williams wrote:
>
> Hi Everyone,
>
> I am running Apache Tomcat 10.1.30 for a web application and notice
~950 timed_waiting threads.
>
> The stack trace for these threads is below:
>
> java.base@17.0.12/
jdk.internal.misc.Unsafe.park(Native<mailto:java.bas
<mailto:java.base@17.0.12/
jdk.internal.misc.Unsafe.park(Native%3cmailto:java.bas>
> e@17.0.12/jdk.internal.misc.Unsafe.park(Native <mailto:e@17.0.12/
jdk.internal.misc.Unsafe.park(Native>> Method);
> java.base@17.0.12/
java.util.concurrent.locks.LockSupport.parkNanos(Loc
<mailto:java.base@17.0.12/
java.util.concurrent.locks.LockSupport.parkNanos(Loc>
>
kSupport.java:252)<mailto:java.base@17.0.12/java.util.concurrent.locks
> .LockSupport.parkNanos(LockSupport.java:252)>;
> java.base@17.0.12/
java.util.concurrent.locks.AbstractQueuedSynchronize
<mailto:java.base@17.0.12/
java.util.concurrent.locks.AbstractQueuedSynchronize>
>
r$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:1674)<mai
>
lto:java.base@17.0.12/java.util.concurrent.locks.AbstractQueuedSynchro
>
nizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:1674)
> >;
> java.base@17.0.12/
java.util.concurrent.LinkedBlockingQueue.poll(Linked
<mailto:java.base@17.0.12/
java.util.concurrent.LinkedBlockingQueue.poll(Linked>
> BlockingQueue.java:460)<mailto:java.base@17.0.12/java.util.concurrent.
> LinkedBlockingQueue.poll(LinkedBlockingQueue.java:460)>;
>
app//org.apache.tomcat.util.threads.TaskQueue.poll(TaskQueue.java:99);
>
app//org.apache.tomcat.util.threads.TaskQueue.poll(TaskQueue.java:33);
>
app//org.apache.tomcat.util.threads.ThreadPoolExecutor.getTask(ThreadP
> oolExecutor.java:1113);
>
app//org.apache.tomcat.util.threads.ThreadPoolExecutor.runWorker(Threa
> dPoolExecutor.java:1175);
>
app//org.apache.tomcat.util.threads.ThreadPoolExecutor$Worker.run(Thre
> adPoolExecutor.java:659);
>
app//org.apache.tomcat.util.threads.TaskThread$WrappingRunnable.run(Ta
> skThread.java:63);
> java.base@17.0.12/
java.lang.Thread.run(Thread.java:840)<mailto:java.ba
<mailto:java.base@17.0.12/
java.lang.Thread.run(Thread.java:840)%3cmailto:java.ba>
> se@17.0.12/java.lang.Thread.run(Thread.java:840)
<mailto:se@17.0.12/ java.lang.Thread.run(Thread.java:840)>>;
>
> I am using an executor with maxThreads set to 1500 and
minSpareThreads set to 64.
>
> I have not seen this issue with Tomcat 10.0.
>
> What could be the issue and how to resolve it? Any help is much
appreciated.
This looks like you have a large number of threads waiting for work.
That seems normal to me if you have a max thread pool of 1500 threads.
What's the problem?
-chris
---------------------------------------------------------------------
To unsubscribe, e-mail: users-unsubscr...@tomcat.apache.org
<mailto:users-unsubscr...@tomcat.apache.org>
For additional commands, e-mail: users-h...@tomcat.apache.org
<mailto:users-h...@tomcat.apache.org>
---------------------------------------------------------------------
To unsubscribe, e-mail: users-unsubscr...@tomcat.apache.org
For additional commands, e-mail: users-h...@tomcat.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: users-unsubscr...@tomcat.apache.org
For additional commands, e-mail: users-h...@tomcat.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: users-unsubscr...@tomcat.apache.org
For additional commands, e-mail: users-h...@tomcat.apache.org
