Hi,
we're running 3 tomcat instances on Solaris. We are experiencing extremely
show shutdowns. I've done a thread dump, but as far as I can tell all looks
ok. Does anyone know what is causing this?
We're calling /opt/ec/tomcat/bin/shutdown.sh to shut down the instance.
I see the line: 'INFO: Stopping Coyote HTTP/1.1 on http-8082' appear
quickly. However the java process hangs about for at least another 3
minutes.
Here's the thread dump:
Dec 15, 2009 7:35:23 AM org.apache.coyote.http11.Http11Protocol pause
INFO: Pausing Coyote HTTP/1.1 on http-8082
Dec 15, 2009 7:35:24 AM org.apache.catalina.core.StandardService stop
INFO: Stopping service Catalina
Dec 15, 2009 7:35:26 AM org.apache.coyote.http11.Http11Protocol destroy
INFO: Stopping Coyote HTTP/1.1 on http-8082
2009-12-15 07:36:20
Full thread dump Java HotSpot(TM) Server VM (11.2-b01 mixed mode):
"DestroyJavaVM" prio=3 tid=0x08070000 nid=0x2 waiting on condition
[0x00000000..0xfe46ebb0]
java.lang.Thread.State: RUNNABLE
"http-8082-1" daemon prio=3 tid=0x0848a800 nid=0xba in Object.wait()
[0xe396d000..0xe396dbf0]
java.lang.Thread.State: WAITING (on object monitor)
at java.lang.Object.wait(Native Method)
at java.lang.Object.wait(Object.java:485)
at
org.apache.tomcat.util.net.JIoEndpoint$Worker.await(JIoEndpoint.java:423)
- locked <0xf9e92aa8> (a
org.apache.tomcat.util.net.JIoEndpoint$Worker)
at
org.apache.tomcat.util.net.JIoEndpoint$Worker.run(JIoEndpoint.java:449)
at java.lang.Thread.run(Thread.java:619)
"Java2D Disposer" daemon prio=3 tid=0x08dd0000 nid=0x50 in Object.wait()
[0xe407b000..0xe407baf0]
java.lang.Thread.State: WAITING (on object monitor)
at java.lang.Object.wait(Native Method)
at java.lang.ref.ReferenceQueue.remove(ReferenceQueue.java:116)
- locked <0xec1c6cc8> (a java.lang.ref.ReferenceQueue$Lock)
at java.lang.ref.ReferenceQueue.remove(ReferenceQueue.java:132)
at sun.java2d.Disposer.run(Disposer.java:125)
at java.lang.Thread.run(Thread.java:619)
"pool-1-thread-10" prio=3 tid=0x085b8800 nid=0x38 waiting on condition
[0xe44d9000..0xe44d9af0]
java.lang.Thread.State: WAITING (parking)
at sun.misc.Unsafe.park(Native Method)
- parking to wait for <0xeb57f170> (a
java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
at java.util.concurrent.locks.LockSupport.park(LockSupport.java:158)
at
java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:1925)
at
java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:358)
at
java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:947)
at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:907)
at java.lang.Thread.run(Thread.java:619)
"pool-1-thread-9" prio=3 tid=0x08e0b400 nid=0x37 waiting on condition
[0xe497d000..0xe497db70]
java.lang.Thread.State: WAITING (parking)
at sun.misc.Unsafe.park(Native Method)
- parking to wait for <0xeb57f170> (a
java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
at java.util.concurrent.locks.LockSupport.park(LockSupport.java:158)
at
java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:1925)
at
java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:358)
at
java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:947)
at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:907)
at java.lang.Thread.run(Thread.java:619)
"pool-1-thread-8" prio=3 tid=0x09550c00 nid=0x36 waiting on condition
[0xe49ce000..0xe49ce9f0]
java.lang.Thread.State: WAITING (parking)
at sun.misc.Unsafe.park(Native Method)
- parking to wait for <0xeb57f170> (a
java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
at java.util.concurrent.locks.LockSupport.park(LockSupport.java:158)
at
java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:1925)
at
java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:358)
at
java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:947)
at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:907)
at java.lang.Thread.run(Thread.java:619)
"pool-1-thread-7" prio=3 tid=0x08e74400 nid=0x35 waiting on condition
[0xe45ae000..0xe45aea70]
java.lang.Thread.State: WAITING (parking)
at sun.misc.Unsafe.park(Native Method)
- parking to wait for <0xeb57f170> (a
java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
at java.util.concurrent.locks.LockSupport.park(LockSupport.java:158)
at
java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:1925)
at
java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:358)
at
java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:947)
at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:907)
at java.lang.Thread.run(Thread.java:619)
"pool-1-thread-6" prio=3 tid=0x08739000 nid=0x34 waiting on condition
[0xe4612000..0xe46128f0]
java.lang.Thread.State: WAITING (parking)
at sun.misc.Unsafe.park(Native Method)
- parking to wait for <0xeb57f170> (a
java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
at java.util.concurrent.locks.LockSupport.park(LockSupport.java:158)
at
java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:1925)
at
java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:358)
at
java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:947)
at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:907)
at java.lang.Thread.run(Thread.java:619)
"pool-1-thread-5" prio=3 tid=0x08b5e400 nid=0x32 waiting on condition
[0xe4663000..0xe4663bf0]
java.lang.Thread.State: WAITING (parking)
at sun.misc.Unsafe.park(Native Method)
- parking to wait for <0xeb57f170> (a
java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
at java.util.concurrent.locks.LockSupport.park(LockSupport.java:158)
at
java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:1925)
at
java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:358)
at
java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:947)
at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:907)
at java.lang.Thread.run(Thread.java:619)
"pool-1-thread-4" prio=3 tid=0x08681400 nid=0x31 waiting on condition
[0xe46b4000..0xe46b4c70]
java.lang.Thread.State: WAITING (parking)
at sun.misc.Unsafe.park(Native Method)
- parking to wait for <0xeb57f170> (a
java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
at java.util.concurrent.locks.LockSupport.park(LockSupport.java:158)
at
java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:1925)
at
java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:358)
at
java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:947)
at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:907)
at java.lang.Thread.run(Thread.java:619)
"MultiThreadedHttpConnectionManager cleanup" daemon prio=3 tid=0x0824a400
nid=0x30 in Object.wait() [0xe4705000..0xe4705af0]
java.lang.Thread.State: WAITING (on object monitor)
at java.lang.Object.wait(Native Method)
at java.lang.ref.ReferenceQueue.remove(ReferenceQueue.java:116)
- locked <0xeb8d2bd8> (a java.lang.ref.ReferenceQueue$Lock)
at java.lang.ref.ReferenceQueue.remove(ReferenceQueue.java:132)
at
org.apache.commons.httpclient.MultiThreadedHttpConnectionManager$ReferenceQueueThread.run(MultiThreadedHttpConnectionManager.java:1122)
"pool-1-thread-3" prio=3 tid=0x08b5f800 nid=0x2f waiting on condition
[0xe4756000..0xe4756b70]
java.lang.Thread.State: WAITING (parking)
at sun.misc.Unsafe.park(Native Method)
- parking to wait for <0xeb57f170> (a
java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
at java.util.concurrent.locks.LockSupport.park(LockSupport.java:158)
at
java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:1925)
at
java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:358)
at
java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:947)
at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:907)
at java.lang.Thread.run(Thread.java:619)
"pool-1-thread-2" prio=3 tid=0x0839f800 nid=0x2d waiting on condition
[0xe47f8000..0xe47f8a70]
java.lang.Thread.State: WAITING (parking)
at sun.misc.Unsafe.park(Native Method)
- parking to wait for <0xeb57f170> (a
java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
at java.util.concurrent.locks.LockSupport.park(LockSupport.java:158)
at
java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:1925)
at
java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:358)
at
java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:947)
at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:907)
at java.lang.Thread.run(Thread.java:619)
"pool-1-thread-1" prio=3 tid=0x087f6c00 nid=0x2a waiting on condition
[0xe4849000..0xe4849bf0]
java.lang.Thread.State: WAITING (parking)
at sun.misc.Unsafe.park(Native Method)
- parking to wait for <0xeb57f170> (a
java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
at java.util.concurrent.locks.LockSupport.park(LockSupport.java:158)
at
java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:1925)
at
java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:358)
at
java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:947)
at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:907)
at java.lang.Thread.run(Thread.java:619)
"pool-3-thread-1" prio=3 tid=0x083fe400 nid=0x29 waiting on condition
[0xe489a000..0xe489ac70]
java.lang.Thread.State: WAITING (parking)
at sun.misc.Unsafe.park(Native Method)
- parking to wait for <0xeb80e2a0> (a
java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
at java.util.concurrent.locks.LockSupport.park(LockSupport.java:158)
at
java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:1925)
at java.util.concurrent.DelayQueue.take(DelayQueue.java:160)
at
java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:583)
at
java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:576)
at
java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:947)
at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:907)
at java.lang.Thread.run(Thread.java:619)
"TP-Processor4" daemon prio=3 tid=0x08a28000 nid=0x1f in Object.wait()
[0xe4ed7000..0xe4ed7b70]
java.lang.Thread.State: WAITING (on object monitor)
at java.lang.Object.wait(Native Method)
at java.lang.Object.wait(Object.java:485)
at org.apache.jk.common.ChannelSocket.accept(ChannelSocket.java:301)
- locked <0xeb4458c8> (a org.apache.jk.common.ChannelSocket)
at
org.apache.jk.common.ChannelSocket.acceptConnections(ChannelSocket.java:661)
at
org.apache.jk.common.ChannelSocket$SocketAcceptor.runIt(ChannelSocket.java:872)
at
org.apache.tomcat.util.threads.ThreadPool$ControlRunnable.run(ThreadPool.java:690)
at java.lang.Thread.run(Thread.java:619)
"http-8082-Acceptor-0" daemon prio=3 tid=0x08535400 nid=0x1b waiting on
condition [0xe5438000..0xe5438970]
java.lang.Thread.State: TIMED_WAITING (sleeping)
at java.lang.Thread.sleep(Native Method)
at
org.apache.tomcat.util.net.JIoEndpoint$Acceptor.run(JIoEndpoint.java:309)
at java.lang.Thread.run(Thread.java:619)
"MySQL Statement Cancellation Timer" daemon prio=3 tid=0x08498400 nid=0x18
in Object.wait() [0xe4f79000..0xe4f79af0]
java.lang.Thread.State: WAITING (on object monitor)
at java.lang.Object.wait(Native Method)
- waiting on <0xeb191718> (a java.util.TaskQueue)
at java.lang.Object.wait(Object.java:485)
at java.util.TimerThread.mainLoop(Timer.java:483)
- locked <0xeb191718> (a java.util.TaskQueue)
at java.util.TimerThread.run(Timer.java:462)
"Low Memory Detector" daemon prio=3 tid=0x081ab400 nid=0x11 runnable
[0x00000000..0x00000000]
java.lang.Thread.State: RUNNABLE
"CompilerThread1" daemon prio=3 tid=0x081a8800 nid=0x10 waiting on condition
[0x00000000..0xe55b2048]
java.lang.Thread.State: RUNNABLE
"CompilerThread0" daemon prio=3 tid=0x081a7000 nid=0xf waiting on condition
[0x00000000..0xe55f30c8]
java.lang.Thread.State: RUNNABLE
"Signal Dispatcher" daemon prio=3 tid=0x081a5c00 nid=0xe waiting on
condition [0x00000000..0x00000000]
java.lang.Thread.State: RUNNABLE
"Finalizer" daemon prio=3 tid=0x08190400 nid=0xd in Object.wait()
[0xe5695000..0xe5695a70]
java.lang.Thread.State: WAITING (on object monitor)
at java.lang.Object.wait(Native Method)
at java.lang.ref.ReferenceQueue.remove(ReferenceQueue.java:116)
- locked <0xeb06e320> (a java.lang.ref.ReferenceQueue$Lock)
at java.lang.ref.ReferenceQueue.remove(ReferenceQueue.java:132)
at java.lang.ref.Finalizer$FinalizerThread.run(Finalizer.java:159)
"Reference Handler" daemon prio=3 tid=0x0818ec00 nid=0xc in Object.wait()
[0xe56e6000..0xe56e68f0]
java.lang.Thread.State: WAITING (on object monitor)
at java.lang.Object.wait(Native Method)
at java.lang.Object.wait(Object.java:485)
at java.lang.ref.Reference$ReferenceHandler.run(Reference.java:116)
- locked <0xeb06b6b8> (a java.lang.ref.Reference$Lock)
"VM Thread" prio=3 tid=0x0818bc00 nid=0xb runnable
"GC task thread#0 (ParallelGC)" prio=3 tid=0x08076800 nid=0x3 runnable
"GC task thread#1 (ParallelGC)" prio=3 tid=0x08077c00 nid=0x4 runnable
"GC task thread#2 (ParallelGC)" prio=3 tid=0x08079000 nid=0x5 runnable
"GC task thread#3 (ParallelGC)" prio=3 tid=0x0807a400 nid=0x6 runnable
"GC task thread#4 (ParallelGC)" prio=3 tid=0x0807b800 nid=0x7 runnable
"GC task thread#5 (ParallelGC)" prio=3 tid=0x0807cc00 nid=0x8 runnable
"GC task thread#6 (ParallelGC)" prio=3 tid=0x0807e000 nid=0x9 runnable
"GC task thread#7 (ParallelGC)" prio=3 tid=0x0807f800 nid=0xa runnable
"VM Periodic Task Thread" prio=3 tid=0x081ad400 nid=0x12 waiting on
condition
JNI global references: 852
Heap
PSYoungGen total 12608K, used 2968K [0xf9200000, 0xfa200000,
0xfb000000)
eden space 8832K, 5% used [0xf9200000,0xf9272180,0xf9aa0000)
from space 3776K, 66% used [0xf9e50000,0xfa0c41e8,0xfa200000)
to space 3776K, 0% used [0xf9aa0000,0xf9aa0000,0xf9e50000)
PSOldGen total 116736K, used 91584K [0xeae00000, 0xf2000000,
0xf9200000)
object space 116736K, 78% used [0xeae00000,0xf0770288,0xf2000000)
PSPermGen total 86016K, used 47160K [0xe5a00000, 0xeae00000,
0xeae00000)
object space 86016K, 54% used [0xe5a00000,0xe880e200,0xeae00000)
many thanks
--
View this message in context:
http://old.nabble.com/Very-slow-shutdown---3-mins-tp26791794p26791794.html
Sent from the Tomcat - User mailing list archive at Nabble.com.
---------------------------------------------------------------------
To unsubscribe, e-mail: users-unsubscr...@tomcat.apache.org
For additional commands, e-mail: users-h...@tomcat.apache.org
