Hi, The issue seems similar to: http://www.tomcatexpert.com/ask-the-experts/tomcat-hangs-production-and-doesn%E2%80%99t-respond-new-http-requests
https://issues.apache.org/bugzilla/show_bug.cgi?id=53173 Regards,Bal. From: balvind...@hotmail.com To: users@tomcat.apache.org Subject: RE: Apache tomcat ( 7.0.19 ) stops processing user requests suddenly but works fine after restart. Date: Tue, 11 Sep 2012 15:04:45 +0000 Hi, I have pasted the first thread dump, and attached 3 other thread dumps. Thread dump1:------------------- 2012-08-10 14:40:52 Full thread dump Java HotSpot(TM) Server VM (11.2-b01 mixed mode): "RMI TCP Connection(23136)-149.184.142.160" - Thread t@37810 java.lang.Thread.State: RUNNABLE at java.net.SocketInputStream.socketRead0(Native Method) at java.net.SocketInputStream.read(Unknown Source) at java.io.BufferedInputStream.fill(Unknown Source) at java.io.BufferedInputStream.read(Unknown Source) - locked java.io.BufferedInputStream@427137 at java.io.FilterInputStream.read(Unknown Source) at sun.rmi.transport.tcp.TCPTransport.handleMessages(Unknown Source) at sun.rmi.transport.tcp.TCPTransport$ConnectionHandler.run0(Unknown Source) at sun.rmi.transport.tcp.TCPTransport$ConnectionHandler.run(Unknown Source) at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(Unknown Source) at java.util.concurrent.ThreadPoolExecutor$Worker.run(Unknown Source) at java.lang.Thread.run(Unknown Source) Locked ownable synchronizers: - locked java.util.concurrent.locks.ReentrantLock$NonfairSync@17119c "JMX server connection timeout 37809" - Thread t@37809 java.lang.Thread.State: TIMED_WAITING on [I@378cf5 at java.lang.Object.wait(Native Method) at com.sun.jmx.remote.internal.ServerCommunicatorAdmin$Timeout.run(Unknown Source) at java.lang.Thread.run(Unknown Source) Locked ownable synchronizers: - None "JMX server connection timeout 37808" - Thread t@37808 java.lang.Thread.State: TIMED_WAITING on [I@1075926 at java.lang.Object.wait(Native Method) at com.sun.jmx.remote.internal.ServerCommunicatorAdmin$Timeout.run(Unknown Source) at java.lang.Thread.run(Unknown Source) Locked ownable synchronizers: - None "RMI TCP Connection(23129)-149.177.35.85" - Thread t@37806 java.lang.Thread.State: RUNNABLE at sun.management.ThreadImpl.dumpThreads0(Native Method) at sun.management.ThreadImpl.dumpAllThreads(Unknown Source) at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) at sun.reflect.NativeMethodAccessorImpl.invoke(Unknown Source) at sun.reflect.DelegatingMethodAccessorImpl.invoke(Unknown Source) at java.lang.reflect.Method.invoke(Unknown Source) at com.sun.jmx.mbeanserver.ConvertingMethod.invokeWithOpenReturn(Unknown Source) at com.sun.jmx.mbeanserver.MXBeanIntrospector.invokeM2(Unknown Source) at com.sun.jmx.mbeanserver.MXBeanIntrospector.invokeM2(Unknown Source) at com.sun.jmx.mbeanserver.MBeanIntrospector.invokeM(Unknown Source) at com.sun.jmx.mbeanserver.PerInterface.invoke(Unknown Source) at com.sun.jmx.mbeanserver.MBeanSupport.invoke(Unknown Source) at javax.management.StandardMBean.invoke(Unknown Source) at com.sun.jmx.interceptor.DefaultMBeanServerInterceptor.invoke(Unknown Source) at com.sun.jmx.mbeanserver.JmxMBeanServer.invoke(Unknown Source) at com.sun.jmx.remote.security.MBeanServerAccessController.invoke(Unknown Source) at javax.management.remote.rmi.RMIConnectionImpl.doOperation(Unknown Source) at javax.management.remote.rmi.RMIConnectionImpl.access$200(Unknown Source) at javax.management.remote.rmi.RMIConnectionImpl$PrivilegedOperation.run(Unknown Source) at java.security.AccessController.doPrivileged(Native Method) at javax.management.remote.rmi.RMIConnectionImpl.doPrivilegedOperation(Unknown Source) at javax.management.remote.rmi.RMIConnectionImpl.invoke(Unknown Source) at sun.reflect.GeneratedMethodAccessor5299.invoke(Unknown Source) at sun.reflect.DelegatingMethodAccessorImpl.invoke(Unknown Source) at java.lang.reflect.Method.invoke(Unknown Source) at sun.rmi.server.UnicastServerRef.dispatch(Unknown Source) at sun.rmi.transport.Transport$1.run(Unknown Source) at java.security.AccessController.doPrivileged(Native Method) at sun.rmi.transport.Transport.serviceCall(Unknown Source) at sun.rmi.transport.tcp.TCPTransport.handleMessages(Unknown Source) at sun.rmi.transport.tcp.TCPTransport$ConnectionHandler.run0(Unknown Source) at sun.rmi.transport.tcp.TCPTransport$ConnectionHandler.run(Unknown Source) at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(Unknown Source) at java.util.concurrent.ThreadPoolExecutor$Worker.run(Unknown Source) at java.lang.Thread.run(Unknown Source) Locked ownable synchronizers: - locked java.util.concurrent.locks.ReentrantLock$NonfairSync@d0793c "RMI TCP Connection(23134)-149.184.142.160" - Thread t@37803 java.lang.Thread.State: RUNNABLE at java.net.SocketInputStream.socketRead0(Native Method) at java.net.SocketInputStream.read(Unknown Source) at java.io.BufferedInputStream.fill(Unknown Source) at java.io.BufferedInputStream.read(Unknown Source) - locked java.io.BufferedInputStream@754035 at java.io.FilterInputStream.read(Unknown Source) at sun.rmi.transport.tcp.TCPTransport.handleMessages(Unknown Source) at sun.rmi.transport.tcp.TCPTransport$ConnectionHandler.run0(Unknown Source) at sun.rmi.transport.tcp.TCPTransport$ConnectionHandler.run(Unknown Source) at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(Unknown Source) at java.util.concurrent.ThreadPoolExecutor$Worker.run(Unknown Source) at java.lang.Thread.run(Unknown Source) Locked ownable synchronizers: - locked java.util.concurrent.locks.ReentrantLock$NonfairSync@1ca727a "RMI TCP Connection(23130)-149.184.142.160" - Thread t@37799 java.lang.Thread.State: RUNNABLE at java.net.SocketInputStream.socketRead0(Native Method) at java.net.SocketInputStream.read(Unknown Source) at java.io.BufferedInputStream.fill(Unknown Source) at java.io.BufferedInputStream.read(Unknown Source) - locked java.io.BufferedInputStream@d3f8d at java.io.FilterInputStream.read(Unknown Source) at sun.rmi.transport.tcp.TCPTransport.handleMessages(Unknown Source) at sun.rmi.transport.tcp.TCPTransport$ConnectionHandler.run0(Unknown Source) at sun.rmi.transport.tcp.TCPTransport$ConnectionHandler.run(Unknown Source) at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(Unknown Source) at java.util.concurrent.ThreadPoolExecutor$Worker.run(Unknown Source) at java.lang.Thread.run(Unknown Source) Locked ownable synchronizers: - locked java.util.concurrent.locks.ReentrantLock$NonfairSync@1c75893 "RMI TCP Connection(23115)-149.184.142.160" - Thread t@37795 java.lang.Thread.State: TIMED_WAITING on com.sun.jmx.remote.internal.ArrayNotificationBuffer@fbfe50 at java.lang.Object.wait(Native Method) at com.sun.jmx.remote.internal.ArrayNotificationBuffer.fetchNotifications(Unknown Source) at com.sun.jmx.remote.internal.ArrayNotificationBuffer$ShareBuffer.fetchNotifications(Unknown Source) at com.sun.jmx.remote.internal.ServerNotifForwarder.fetchNotifs(Unknown Source) at javax.management.remote.rmi.RMIConnectionImpl$2.run(Unknown Source) at javax.management.remote.rmi.RMIConnectionImpl$2.run(Unknown Source) at java.security.AccessController.doPrivileged(Native Method) at javax.management.remote.rmi.RMIConnectionImpl.fetchNotifications(Unknown Source) at sun.reflect.GeneratedMethodAccessor31.invoke(Unknown Source) at sun.reflect.DelegatingMethodAccessorImpl.invoke(Unknown Source) at java.lang.reflect.Method.invoke(Unknown Source) at sun.rmi.server.UnicastServerRef.dispatch(Unknown Source) at sun.rmi.transport.Transport$1.run(Unknown Source) at java.security.AccessController.doPrivileged(Native Method) at sun.rmi.transport.Transport.serviceCall(Unknown Source) at sun.rmi.transport.tcp.TCPTransport.handleMessages(Unknown Source) at sun.rmi.transport.tcp.TCPTransport$ConnectionHandler.run0(Unknown Source) at sun.rmi.transport.tcp.TCPTransport$ConnectionHandler.run(Unknown Source) at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(Unknown Source) at java.util.concurrent.ThreadPoolExecutor$Worker.run(Unknown Source) at java.lang.Thread.run(Unknown Source) Locked ownable synchronizers: - locked java.util.concurrent.locks.ReentrantLock$NonfairSync@88281d "RMI TCP Connection(23135)-149.184.142.160" - Thread t@37787 java.lang.Thread.State: RUNNABLE at java.net.SocketInputStream.socketRead0(Native Method) at java.net.SocketInputStream.read(Unknown Source) at java.io.BufferedInputStream.fill(Unknown Source) at java.io.BufferedInputStream.read(Unknown Source) - locked java.io.BufferedInputStream@9d3697 at java.io.FilterInputStream.read(Unknown Source) at sun.rmi.transport.tcp.TCPTransport.handleMessages(Unknown Source) at sun.rmi.transport.tcp.TCPTransport$ConnectionHandler.run0(Unknown Source) at sun.rmi.transport.tcp.TCPTransport$ConnectionHandler.run(Unknown Source) at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(Unknown Source) at java.util.concurrent.ThreadPoolExecutor$Worker.run(Unknown Source) at java.lang.Thread.run(Unknown Source) Locked ownable synchronizers: - locked java.util.concurrent.locks.ReentrantLock$NonfairSync@16af9f9 "RMI TCP Connection(23062)-149.184.142.160" - Thread t@37756 java.lang.Thread.State: TIMED_WAITING on com.sun.jmx.remote.internal.ArrayNotificationBuffer@fbfe50 at java.lang.Object.wait(Native Method) at com.sun.jmx.remote.internal.ArrayNotificationBuffer.fetchNotifications(Unknown Source) at com.sun.jmx.remote.internal.ArrayNotificationBuffer$ShareBuffer.fetchNotifications(Unknown Source) at com.sun.jmx.remote.internal.ServerNotifForwarder.fetchNotifs(Unknown Source) at javax.management.remote.rmi.RMIConnectionImpl$2.run(Unknown Source) at javax.management.remote.rmi.RMIConnectionImpl$2.run(Unknown Source) at java.security.AccessController.doPrivileged(Native Method) at javax.management.remote.rmi.RMIConnectionImpl.fetchNotifications(Unknown Source) at sun.reflect.GeneratedMethodAccessor31.invoke(Unknown Source) at sun.reflect.DelegatingMethodAccessorImpl.invoke(Unknown Source) at java.lang.reflect.Method.invoke(Unknown Source) at sun.rmi.server.UnicastServerRef.dispatch(Unknown Source) at sun.rmi.transport.Transport$1.run(Unknown Source) at java.security.AccessController.doPrivileged(Native Method) at sun.rmi.transport.Transport.serviceCall(Unknown Source) at sun.rmi.transport.tcp.TCPTransport.handleMessages(Unknown Source) at sun.rmi.transport.tcp.TCPTransport$ConnectionHandler.run0(Unknown Source) at sun.rmi.transport.tcp.TCPTransport$ConnectionHandler.run(Unknown Source) at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(Unknown Source) at java.util.concurrent.ThreadPoolExecutor$Worker.run(Unknown Source) at java.lang.Thread.run(Unknown Source) Locked ownable synchronizers: - locked java.util.concurrent.locks.ReentrantLock$NonfairSync@415da0 "http-bio-8443-exec-13182" - Thread t@37623 java.lang.Thread.State: RUNNABLE at java.net.SocketInputStream.socketRead0(Native Method) at java.net.SocketInputStream.read(Unknown Source) at com.sun.net.ssl.internal.ssl.InputRecord.readFully(Unknown Source) at com.sun.net.ssl.internal.ssl.InputRecord.read(Unknown Source) at com.sun.net.ssl.internal.ssl.SSLSocketImpl.readRecord(Unknown Source) - locked java.lang.Object@169bcbd at com.sun.net.ssl.internal.ssl.SSLSocketImpl.readDataRecord(Unknown Source) at com.sun.net.ssl.internal.ssl.AppInputStream.read(Unknown Source) - locked com.sun.net.ssl.internal.ssl.AppInputStream@e61d04 at org.apache.coyote.http11.InternalInputBuffer.fill(InternalInputBuffer.java:445) at org.apache.coyote.http11.InternalInputBuffer.fill(InternalInputBuffer.java:430) at org.apache.coyote.http11.Http11Processor.process(Http11Processor.java:215) at org.apache.coyote.AbstractProtocol$AbstractConnectionHandler.process(AbstractProtocol.java:515) at org.apache.tomcat.util.net.JIoEndpoint$SocketProcessor.run(JIoEndpoint.java:302) - locked org.apache.tomcat.util.net.SocketWrapper@17b14f2 at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(Unknown Source) at java.util.concurrent.ThreadPoolExecutor$Worker.run(Unknown Source) at java.lang.Thread.run(Unknown Source) Locked ownable synchronizers: - locked java.util.concurrent.locks.ReentrantLock$NonfairSync@19c4cd7 "http-bio-8443-exec-13181" - Thread t@37619 java.lang.Thread.State: WAITING on java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@1f31168 at sun.misc.Unsafe.park(Native Method) at java.util.concurrent.locks.LockSupport.park(Unknown Source) at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(Unknown Source) at java.util.concurrent.LinkedBlockingQueue.take(Unknown Source) at org.apache.tomcat.util.threads.TaskQueue.take(TaskQueue.java:104) at org.apache.tomcat.util.threads.TaskQueue.take(TaskQueue.java:32) at java.util.concurrent.ThreadPoolExecutor.getTask(Unknown Source) at java.util.concurrent.ThreadPoolExecutor$Worker.run(Unknown Source) at java.lang.Thread.run(Unknown Source) Locked ownable synchronizers: - None "http-bio-8443-exec-13180" - Thread t@37618 java.lang.Thread.State: WAITING on java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@1f31168 at sun.misc.Unsafe.park(Native Method) at java.util.concurrent.locks.LockSupport.park(Unknown Source) at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(Unknown Source) at java.util.concurrent.LinkedBlockingQueue.take(Unknown Source) at org.apache.tomcat.util.threads.TaskQueue.take(TaskQueue.java:104) at org.apache.tomcat.util.threads.TaskQueue.take(TaskQueue.java:32) at java.util.concurrent.ThreadPoolExecutor.getTask(Unknown Source) at java.util.concurrent.ThreadPoolExecutor$Worker.run(Unknown Source) at java.lang.Thread.run(Unknown Source) Locked ownable synchronizers: - None "http-bio-8443-exec-13179" - Thread t@37615 java.lang.Thread.State: WAITING on java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@1f31168 at sun.misc.Unsafe.park(Native Method) at java.util.concurrent.locks.LockSupport.park(Unknown Source) at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(Unknown Source) at java.util.concurrent.LinkedBlockingQueue.take(Unknown Source) at org.apache.tomcat.util.threads.TaskQueue.take(TaskQueue.java:104) at org.apache.tomcat.util.threads.TaskQueue.take(TaskQueue.java:32) at java.util.concurrent.ThreadPoolExecutor.getTask(Unknown Source) at java.util.concurrent.ThreadPoolExecutor$Worker.run(Unknown Source) at java.lang.Thread.run(Unknown Source) Locked ownable synchronizers: - None "http-bio-8443-exec-13176" - Thread t@37584 java.lang.Thread.State: RUNNABLE at java.net.SocketInputStream.socketRead0(Native Method) at java.net.SocketInputStream.read(Unknown Source) at com.sun.net.ssl.internal.ssl.InputRecord.readFully(Unknown Source) at com.sun.net.ssl.internal.ssl.InputRecord.read(Unknown Source) at com.sun.net.ssl.internal.ssl.SSLSocketImpl.readRecord(Unknown Source) - locked java.lang.Object@9d2f3 at com.sun.net.ssl.internal.ssl.SSLSocketImpl.readDataRecord(Unknown Source) at com.sun.net.ssl.internal.ssl.AppInputStream.read(Unknown Source) - locked com.sun.net.ssl.internal.ssl.AppInputStream@f8902a at org.apache.coyote.http11.InternalInputBuffer.fill(InternalInputBuffer.java:445) at org.apache.coyote.http11.InternalInputBuffer.fill(InternalInputBuffer.java:430) at org.apache.coyote.http11.Http11Processor.process(Http11Processor.java:215) at org.apache.coyote.AbstractProtocol$AbstractConnectionHandler.process(AbstractProtocol.java:515) at org.apache.tomcat.util.net.JIoEndpoint$SocketProcessor.run(JIoEndpoint.java:302) - locked org.apache.tomcat.util.net.SocketWrapper@c0d197 at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(Unknown Source) at java.util.concurrent.ThreadPoolExecutor$Worker.run(Unknown Source) at java.lang.Thread.run(Unknown Source) Locked ownable synchronizers: - locked java.util.concurrent.locks.ReentrantLock$NonfairSync@1004e70 "http-bio-8443-exec-13175" - Thread t@37583 java.lang.Thread.State: WAITING on java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@1f31168 at sun.misc.Unsafe.park(Native Method) at java.util.concurrent.locks.LockSupport.park(Unknown Source) at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(Unknown Source) at java.util.concurrent.LinkedBlockingQueue.take(Unknown Source) at org.apache.tomcat.util.threads.TaskQueue.take(TaskQueue.java:104) at org.apache.tomcat.util.threads.TaskQueue.take(TaskQueue.java:32) at java.util.concurrent.ThreadPoolExecutor.getTask(Unknown Source) at java.util.concurrent.ThreadPoolExecutor$Worker.run(Unknown Source) at java.lang.Thread.run(Unknown Source) Locked ownable synchronizers: - None "http-bio-8443-exec-13174" - Thread t@37582 java.lang.Thread.State: WAITING on java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@1f31168 at sun.misc.Unsafe.park(Native Method) at java.util.concurrent.locks.LockSupport.park(Unknown Source) at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(Unknown Source) at java.util.concurrent.LinkedBlockingQueue.take(Unknown Source) at org.apache.tomcat.util.threads.TaskQueue.take(TaskQueue.java:104) at org.apache.tomcat.util.threads.TaskQueue.take(TaskQueue.java:32) at java.util.concurrent.ThreadPoolExecutor.getTask(Unknown Source) at java.util.concurrent.ThreadPoolExecutor$Worker.run(Unknown Source) at java.lang.Thread.run(Unknown Source) Locked ownable synchronizers: - None "http-bio-8443-exec-13172" - Thread t@37580 java.lang.Thread.State: WAITING on java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@1f31168 at sun.misc.Unsafe.park(Native Method) at java.util.concurrent.locks.LockSupport.park(Unknown Source) at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(Unknown Source) at java.util.concurrent.LinkedBlockingQueue.take(Unknown Source) at org.apache.tomcat.util.threads.TaskQueue.take(TaskQueue.java:104) at org.apache.tomcat.util.threads.TaskQueue.take(TaskQueue.java:32) at java.util.concurrent.ThreadPoolExecutor.getTask(Unknown Source) at java.util.concurrent.ThreadPoolExecutor$Worker.run(Unknown Source) at java.lang.Thread.run(Unknown Source) Locked ownable synchronizers: - None "http-bio-8443-exec-13157" - Thread t@37493 java.lang.Thread.State: RUNNABLE at java.net.SocketInputStream.socketRead0(Native Method) at java.net.SocketInputStream.read(Unknown Source) at com.sun.net.ssl.internal.ssl.InputRecord.readFully(Unknown Source) at com.sun.net.ssl.internal.ssl.InputRecord.read(Unknown Source) at com.sun.net.ssl.internal.ssl.SSLSocketImpl.readRecord(Unknown Source) - locked java.lang.Object@11db953 at com.sun.net.ssl.internal.ssl.SSLSocketImpl.readDataRecord(Unknown Source) at com.sun.net.ssl.internal.ssl.AppInputStream.read(Unknown Source) - locked com.sun.net.ssl.internal.ssl.AppInputStream@7200ec at org.apache.coyote.http11.InternalInputBuffer.fill(InternalInputBuffer.java:445) at org.apache.coyote.http11.InternalInputBuffer.fill(InternalInputBuffer.java:430) at org.apache.coyote.http11.Http11Processor.process(Http11Processor.java:215) at org.apache.coyote.AbstractProtocol$AbstractConnectionHandler.process(AbstractProtocol.java:515) at org.apache.tomcat.util.net.JIoEndpoint$SocketProcessor.run(JIoEndpoint.java:302) - locked org.apache.tomcat.util.net.SocketWrapper@9aa1e0 at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(Unknown Source) at java.util.concurrent.ThreadPoolExecutor$Worker.run(Unknown Source) at java.lang.Thread.run(Unknown Source) Locked ownable synchronizers: - locked java.util.concurrent.locks.ReentrantLock$NonfairSync@1f9b5f5 "http-bio-8443-exec-13153" - Thread t@37489 java.lang.Thread.State: RUNNABLE at java.net.SocketInputStream.socketRead0(Native Method) at java.net.SocketInputStream.read(Unknown Source) at com.sun.net.ssl.internal.ssl.InputRecord.readFully(Unknown Source) at com.sun.net.ssl.internal.ssl.InputRecord.read(Unknown Source) at com.sun.net.ssl.internal.ssl.SSLSocketImpl.readRecord(Unknown Source) - locked java.lang.Object@118f940 at com.sun.net.ssl.internal.ssl.SSLSocketImpl.readDataRecord(Unknown Source) at com.sun.net.ssl.internal.ssl.AppInputStream.read(Unknown Source) - locked com.sun.net.ssl.internal.ssl.AppInputStream@93e184 at org.apache.coyote.http11.InternalInputBuffer.fill(InternalInputBuffer.java:445) at org.apache.coyote.http11.InternalInputBuffer.fill(InternalInputBuffer.java:430) at org.apache.coyote.http11.Http11Processor.process(Http11Processor.java:215) at org.apache.coyote.AbstractProtocol$AbstractConnectionHandler.process(AbstractProtocol.java:515) at org.apache.tomcat.util.net.JIoEndpoint$SocketProcessor.run(JIoEndpoint.java:302) - locked org.apache.tomcat.util.net.SocketWrapper@bae208 at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(Unknown Source) at java.util.concurrent.ThreadPoolExecutor$Worker.run(Unknown Source) at java.lang.Thread.run(Unknown Source) Locked ownable synchronizers: - locked java.util.concurrent.locks.ReentrantLock$NonfairSync@14d8afa "JMX server connection timeout 20278" - Thread t@20278 java.lang.Thread.State: TIMED_WAITING on [I@172a3d4 at java.lang.Object.wait(Native Method) at com.sun.jmx.remote.internal.ServerCommunicatorAdmin$Timeout.run(Unknown Source) at java.lang.Thread.run(Unknown Source) Locked ownable synchronizers: - None "JMX server connection timeout 20277" - Thread t@20277 java.lang.Thread.State: TIMED_WAITING on [I@161b40a at java.lang.Object.wait(Native Method) at com.sun.jmx.remote.internal.ServerCommunicatorAdmin$Timeout.run(Unknown Source) at java.lang.Thread.run(Unknown Source) Locked ownable synchronizers: - None "TIBCO EMS TCPLink Reader (Server-8211461)" - Thread t@14697 java.lang.Thread.State: RUNNABLE at java.net.SocketInputStream.socketRead0(Native Method) at java.net.SocketInputStream.read(Unknown Source) at com.sun.net.ssl.internal.ssl.InputRecord.readFully(Unknown Source) at com.sun.net.ssl.internal.ssl.InputRecord.read(Unknown Source) at com.sun.net.ssl.internal.ssl.SSLSocketImpl.readRecord(Unknown Source) - locked java.lang.Object@15c19ae at com.sun.net.ssl.internal.ssl.SSLSocketImpl.readDataRecord(Unknown Source) at com.sun.net.ssl.internal.ssl.AppInputStream.read(Unknown Source) - locked com.sun.net.ssl.internal.ssl.AppInputStream@199ff9e at java.io.BufferedInputStream.fill(Unknown Source) at java.io.BufferedInputStream.read(Unknown Source) - locked java.io.BufferedInputStream@15cf15b at java.io.DataInputStream.readInt(Unknown Source) at com.tibco.tibjms.TibjmsxLinkTcp._readWireMsg(TibjmsxLinkTcp.java:629) at com.tibco.tibjms.TibjmsxLinkSSL._readWireMsg(TibjmsxLinkSSL.java:651) at com.tibco.tibjms.TibjmsxLinkTcp$LinkReader.work(TibjmsxLinkTcp.java:280) at com.tibco.tibjms.TibjmsxLinkTcp$LinkReader.run(TibjmsxLinkTcp.java:259) Locked ownable synchronizers: - None "TIBCO EMS TCPLink Reader (Server-7262243)" - Thread t@6265 java.lang.Thread.State: RUNNABLE at java.net.SocketInputStream.socketRead0(Native Method) at java.net.SocketInputStream.read(Unknown Source) at com.sun.net.ssl.internal.ssl.InputRecord.readFully(Unknown Source) at com.sun.net.ssl.internal.ssl.InputRecord.read(Unknown Source) at com.sun.net.ssl.internal.ssl.SSLSocketImpl.readRecord(Unknown Source) - locked java.lang.Object@2e0c34 at com.sun.net.ssl.internal.ssl.SSLSocketImpl.readDataRecord(Unknown Source) at com.sun.net.ssl.internal.ssl.AppInputStream.read(Unknown Source) - locked com.sun.net.ssl.internal.ssl.AppInputStream@1233759 at java.io.BufferedInputStream.fill(Unknown Source) at java.io.BufferedInputStream.read(Unknown Source) - locked java.io.BufferedInputStream@106d785 at java.io.DataInputStream.readInt(Unknown Source) at com.tibco.tibjms.TibjmsxLinkTcp._readWireMsg(TibjmsxLinkTcp.java:629) at com.tibco.tibjms.TibjmsxLinkSSL._readWireMsg(TibjmsxLinkSSL.java:651) at com.tibco.tibjms.TibjmsxLinkTcp$LinkReader.work(TibjmsxLinkTcp.java:280) at com.tibco.tibjms.TibjmsxLinkTcp$LinkReader.run(TibjmsxLinkTcp.java:259) Locked ownable synchronizers: - None "TIBCO EMS TCPLink Reader (Server-34701031)" - Thread t@4657 java.lang.Thread.State: RUNNABLE at java.net.SocketInputStream.socketRead0(Native Method) at java.net.SocketInputStream.read(Unknown Source) at com.sun.net.ssl.internal.ssl.InputRecord.readFully(Unknown Source) at com.sun.net.ssl.internal.ssl.InputRecord.read(Unknown Source) at com.sun.net.ssl.internal.ssl.SSLSocketImpl.readRecord(Unknown Source) - locked java.lang.Object@2f626a at com.sun.net.ssl.internal.ssl.SSLSocketImpl.readDataRecord(Unknown Source) at com.sun.net.ssl.internal.ssl.AppInputStream.read(Unknown Source) - locked com.sun.net.ssl.internal.ssl.AppInputStream@e655a9 at java.io.BufferedInputStream.fill(Unknown Source) at java.io.BufferedInputStream.read(Unknown Source) - locked java.io.BufferedInputStream@2000c at java.io.DataInputStream.readInt(Unknown Source) at com.tibco.tibjms.TibjmsxLinkTcp._readWireMsg(TibjmsxLinkTcp.java:629) at com.tibco.tibjms.TibjmsxLinkSSL._readWireMsg(TibjmsxLinkSSL.java:651) at com.tibco.tibjms.TibjmsxLinkTcp$LinkReader.work(TibjmsxLinkTcp.java:280) at com.tibco.tibjms.TibjmsxLinkTcp$LinkReader.run(TibjmsxLinkTcp.java:259) Locked ownable synchronizers: - None "TIBCO EMS TCPLink Reader (Server-10500556)" - Thread t@4580 java.lang.Thread.State: RUNNABLE at java.net.SocketInputStream.socketRead0(Native Method) at java.net.SocketInputStream.read(Unknown Source) at com.sun.net.ssl.internal.ssl.InputRecord.readFully(Unknown Source) at com.sun.net.ssl.internal.ssl.InputRecord.read(Unknown Source) at com.sun.net.ssl.internal.ssl.SSLSocketImpl.readRecord(Unknown Source) - locked java.lang.Object@1514a33 at com.sun.net.ssl.internal.ssl.SSLSocketImpl.readDataRecord(Unknown Source) at com.sun.net.ssl.internal.ssl.AppInputStream.read(Unknown Source) - locked com.sun.net.ssl.internal.ssl.AppInputStream@1de6f76 at java.io.BufferedInputStream.fill(Unknown Source) at java.io.BufferedInputStream.read(Unknown Source) - locked java.io.BufferedInputStream@67b838 at java.io.DataInputStream.readInt(Unknown Source) at com.tibco.tibjms.TibjmsxLinkTcp._readWireMsg(TibjmsxLinkTcp.java:629) at com.tibco.tibjms.TibjmsxLinkSSL._readWireMsg(TibjmsxLinkSSL.java:651) at com.tibco.tibjms.TibjmsxLinkTcp$LinkReader.work(TibjmsxLinkTcp.java:280) at com.tibco.tibjms.TibjmsxLinkTcp$LinkReader.run(TibjmsxLinkTcp.java:259) Locked ownable synchronizers: - None "TIBCO EMS TCPLink Reader (Server-10479806)" - Thread t@3911 java.lang.Thread.State: RUNNABLE at java.net.SocketInputStream.socketRead0(Native Method) at java.net.SocketInputStream.read(Unknown Source) at com.sun.net.ssl.internal.ssl.InputRecord.readFully(Unknown Source) at com.sun.net.ssl.internal.ssl.InputRecord.read(Unknown Source) at com.sun.net.ssl.internal.ssl.SSLSocketImpl.readRecord(Unknown Source) - locked java.lang.Object@16f6ae4 at com.sun.net.ssl.internal.ssl.SSLSocketImpl.readDataRecord(Unknown Source) at com.sun.net.ssl.internal.ssl.AppInputStream.read(Unknown Source) - locked com.sun.net.ssl.internal.ssl.AppInputStream@1d2d6dd at java.io.BufferedInputStream.fill(Unknown Source) at java.io.BufferedInputStream.read(Unknown Source) - locked java.io.BufferedInputStream@1185eab at java.io.DataInputStream.readInt(Unknown Source) at com.tibco.tibjms.TibjmsxLinkTcp._readWireMsg(TibjmsxLinkTcp.java:629) at com.tibco.tibjms.TibjmsxLinkSSL._readWireMsg(TibjmsxLinkSSL.java:651) at com.tibco.tibjms.TibjmsxLinkTcp$LinkReader.work(TibjmsxLinkTcp.java:280) at com.tibco.tibjms.TibjmsxLinkTcp$LinkReader.run(TibjmsxLinkTcp.java:259) Locked ownable synchronizers: - None "TIBCO EMS TCPLink Reader (Server-7530293)" - Thread t@3746 java.lang.Thread.State: RUNNABLE at java.net.SocketInputStream.socketRead0(Native Method) at java.net.SocketInputStream.read(Unknown Source) at com.sun.net.ssl.internal.ssl.InputRecord.readFully(Unknown Source) at com.sun.net.ssl.internal.ssl.InputRecord.read(Unknown Source) at com.sun.net.ssl.internal.ssl.SSLSocketImpl.readRecord(Unknown Source) - locked java.lang.Object@ef6481 at com.sun.net.ssl.internal.ssl.SSLSocketImpl.readDataRecord(Unknown Source) at com.sun.net.ssl.internal.ssl.AppInputStream.read(Unknown Source) - locked com.sun.net.ssl.internal.ssl.AppInputStream@af7fcb at java.io.BufferedInputStream.fill(Unknown Source) at java.io.BufferedInputStream.read(Unknown Source) - locked java.io.BufferedInputStream@1eef61c at java.io.DataInputStream.readInt(Unknown Source) at com.tibco.tibjms.TibjmsxLinkTcp._readWireMsg(TibjmsxLinkTcp.java:629) at com.tibco.tibjms.TibjmsxLinkSSL._readWireMsg(TibjmsxLinkSSL.java:651) at com.tibco.tibjms.TibjmsxLinkTcp$LinkReader.work(TibjmsxLinkTcp.java:280) at com.tibco.tibjms.TibjmsxLinkTcp$LinkReader.run(TibjmsxLinkTcp.java:259) Locked ownable synchronizers: - None "TIBCO EMS TCPLink Reader (Server-7208323)" - Thread t@3705 java.lang.Thread.State: RUNNABLE at java.net.SocketInputStream.socketRead0(Native Method) at java.net.SocketInputStream.read(Unknown Source) at com.sun.net.ssl.internal.ssl.InputRecord.readFully(Unknown Source) at com.sun.net.ssl.internal.ssl.InputRecord.read(Unknown Source) at com.sun.net.ssl.internal.ssl.SSLSocketImpl.readRecord(Unknown Source) - locked java.lang.Object@6966e3 at com.sun.net.ssl.internal.ssl.SSLSocketImpl.readDataRecord(Unknown Source) at com.sun.net.ssl.internal.ssl.AppInputStream.read(Unknown Source) - locked com.sun.net.ssl.internal.ssl.AppInputStream@1e1e638 at java.io.BufferedInputStream.fill(Unknown Source) at java.io.BufferedInputStream.read(Unknown Source) - locked java.io.BufferedInputStream@1d1a81f at java.io.DataInputStream.readInt(Unknown Source) at com.tibco.tibjms.TibjmsxLinkTcp._readWireMsg(TibjmsxLinkTcp.java:629) at com.tibco.tibjms.TibjmsxLinkSSL._readWireMsg(TibjmsxLinkSSL.java:651) at com.tibco.tibjms.TibjmsxLinkTcp$LinkReader.work(TibjmsxLinkTcp.java:280) at com.tibco.tibjms.TibjmsxLinkTcp$LinkReader.run(TibjmsxLinkTcp.java:259) Locked ownable synchronizers: - None "TIBCO EMS Connections Pinger" - Thread t@3704 java.lang.Thread.State: TIMED_WAITING at java.lang.Thread.sleep(Native Method) at com.tibco.tibjms.TibjmsxLinkTcp$Pinger.run(TibjmsxLinkTcp.java:154) at java.lang.Thread.run(Unknown Source) Locked ownable synchronizers: - None "Timer-1" - Thread t@3630 java.lang.Thread.State: TIMED_WAITING on java.util.TaskQueue@1f564da at java.lang.Object.wait(Native Method) at java.util.TimerThread.mainLoop(Unknown Source) at java.util.TimerThread.run(Unknown Source) Locked ownable synchronizers: - None "TIBCO EMS TCPLink Reader (Server-7291923)" - Thread t@677 java.lang.Thread.State: RUNNABLE at java.net.SocketInputStream.socketRead0(Native Method) at java.net.SocketInputStream.read(Unknown Source) at com.sun.net.ssl.internal.ssl.InputRecord.readFully(Unknown Source) at com.sun.net.ssl.internal.ssl.InputRecord.read(Unknown Source) at com.sun.net.ssl.internal.ssl.SSLSocketImpl.readRecord(Unknown Source) - locked java.lang.Object@174b00f at com.sun.net.ssl.internal.ssl.SSLSocketImpl.readDataRecord(Unknown Source) at com.sun.net.ssl.internal.ssl.AppInputStream.read(Unknown Source) - locked com.sun.net.ssl.internal.ssl.AppInputStream@1d2b3ca at java.io.BufferedInputStream.fill(Unknown Source) at java.io.BufferedInputStream.read(Unknown Source) - locked java.io.BufferedInputStream@19a318 at java.io.DataInputStream.readInt(Unknown Source) at com.tibco.tibjms.TibjmsxLinkTcp._readWireMsg(TibjmsxLinkTcp.java:629) at com.tibco.tibjms.TibjmsxLinkSSL._readWireMsg(TibjmsxLinkSSL.java:651) at com.tibco.tibjms.TibjmsxLinkTcp$LinkReader.work(TibjmsxLinkTcp.java:280) at com.tibco.tibjms.TibjmsxLinkTcp$LinkReader.run(TibjmsxLinkTcp.java:259) Locked ownable synchronizers: - None "TIBCO EMS TCPLink Reader (Server-7097900)" - Thread t@380 java.lang.Thread.State: RUNNABLE at java.net.SocketInputStream.socketRead0(Native Method) at java.net.SocketInputStream.read(Unknown Source) at com.sun.net.ssl.internal.ssl.InputRecord.readFully(Unknown Source) at com.sun.net.ssl.internal.ssl.InputRecord.read(Unknown Source) at com.sun.net.ssl.internal.ssl.SSLSocketImpl.readRecord(Unknown Source) - locked java.lang.Object@1229e8f at com.sun.net.ssl.internal.ssl.SSLSocketImpl.readDataRecord(Unknown Source) at com.sun.net.ssl.internal.ssl.AppInputStream.read(Unknown Source) - locked com.sun.net.ssl.internal.ssl.AppInputStream@1d7cf47 at java.io.BufferedInputStream.fill(Unknown Source) at java.io.BufferedInputStream.read(Unknown Source) - locked java.io.BufferedInputStream@1697394 at java.io.DataInputStream.readInt(Unknown Source) at com.tibco.tibjms.TibjmsxLinkTcp._readWireMsg(TibjmsxLinkTcp.java:629) at com.tibco.tibjms.TibjmsxLinkSSL._readWireMsg(TibjmsxLinkSSL.java:651) at com.tibco.tibjms.TibjmsxLinkTcp$LinkReader.work(TibjmsxLinkTcp.java:280) at com.tibco.tibjms.TibjmsxLinkTcp$LinkReader.run(TibjmsxLinkTcp.java:259) Locked ownable synchronizers: - None "TIBCO EMS TCPLink Reader (Server-10324836)" - Thread t@157 java.lang.Thread.State: RUNNABLE at java.net.SocketInputStream.socketRead0(Native Method) at java.net.SocketInputStream.read(Unknown Source) at com.sun.net.ssl.internal.ssl.InputRecord.readFully(Unknown Source) at com.sun.net.ssl.internal.ssl.InputRecord.read(Unknown Source) at com.sun.net.ssl.internal.ssl.SSLSocketImpl.readRecord(Unknown Source) - locked java.lang.Object@22fe5f at com.sun.net.ssl.internal.ssl.SSLSocketImpl.readDataRecord(Unknown Source) at com.sun.net.ssl.internal.ssl.AppInputStream.read(Unknown Source) - locked com.sun.net.ssl.internal.ssl.AppInputStream@16a4569 at java.io.BufferedInputStream.fill(Unknown Source) at java.io.BufferedInputStream.read(Unknown Source) - locked java.io.BufferedInputStream@1f3009b at java.io.DataInputStream.readInt(Unknown Source) at com.tibco.tibjms.TibjmsxLinkTcp._readWireMsg(TibjmsxLinkTcp.java:629) at com.tibco.tibjms.TibjmsxLinkSSL._readWireMsg(TibjmsxLinkSSL.java:651) at com.tibco.tibjms.TibjmsxLinkTcp$LinkReader.work(TibjmsxLinkTcp.java:280) at com.tibco.tibjms.TibjmsxLinkTcp$LinkReader.run(TibjmsxLinkTcp.java:259) Locked ownable synchronizers: - None "Timer-0" - Thread t@155 java.lang.Thread.State: TIMED_WAITING on java.util.TaskQueue@1b0504c at java.lang.Object.wait(Native Method) at java.util.TimerThread.mainLoop(Unknown Source) at java.util.TimerThread.run(Unknown Source) Locked ownable synchronizers: - None "TIBCO EMS TCPLink Reader (Server-34276374)" - Thread t@97 java.lang.Thread.State: RUNNABLE at java.net.SocketInputStream.socketRead0(Native Method) at java.net.SocketInputStream.read(Unknown Source) at com.sun.net.ssl.internal.ssl.InputRecord.readFully(Unknown Source) at com.sun.net.ssl.internal.ssl.InputRecord.read(Unknown Source) at com.sun.net.ssl.internal.ssl.SSLSocketImpl.readRecord(Unknown Source) - locked java.lang.Object@acfea9 at com.sun.net.ssl.internal.ssl.SSLSocketImpl.readDataRecord(Unknown Source) at com.sun.net.ssl.internal.ssl.AppInputStream.read(Unknown Source) - locked com.sun.net.ssl.internal.ssl.AppInputStream@1832a6b at java.io.BufferedInputStream.fill(Unknown Source) at java.io.BufferedInputStream.read(Unknown Source) - locked java.io.BufferedInputStream@55e5f0 at java.io.DataInputStream.readInt(Unknown Source) at com.tibco.tibjms.TibjmsxLinkTcp._readWireMsg(TibjmsxLinkTcp.java:629) at com.tibco.tibjms.TibjmsxLinkSSL._readWireMsg(TibjmsxLinkSSL.java:651) at com.tibco.tibjms.TibjmsxLinkTcp$LinkReader.work(TibjmsxLinkTcp.java:280) at com.tibco.tibjms.TibjmsxLinkTcp$LinkReader.run(TibjmsxLinkTcp.java:259) Locked ownable synchronizers: - None "TIBCO EMS TCPLink Reader (Server-7093946)" - Thread t@91 java.lang.Thread.State: RUNNABLE at java.net.SocketInputStream.socketRead0(Native Method) at java.net.SocketInputStream.read(Unknown Source) at com.sun.net.ssl.internal.ssl.InputRecord.readFully(Unknown Source) at com.sun.net.ssl.internal.ssl.InputRecord.read(Unknown Source) at com.sun.net.ssl.internal.ssl.SSLSocketImpl.readRecord(Unknown Source) - locked java.lang.Object@16af6ea at com.sun.net.ssl.internal.ssl.SSLSocketImpl.readDataRecord(Unknown Source) at com.sun.net.ssl.internal.ssl.AppInputStream.read(Unknown Source) - locked com.sun.net.ssl.internal.ssl.AppInputStream@17891e4 at java.io.BufferedInputStream.fill(Unknown Source) at java.io.BufferedInputStream.read(Unknown Source) - locked java.io.BufferedInputStream@b2270e at java.io.DataInputStream.readInt(Unknown Source) at com.tibco.tibjms.TibjmsxLinkTcp._readWireMsg(TibjmsxLinkTcp.java:629) at com.tibco.tibjms.TibjmsxLinkSSL._readWireMsg(TibjmsxLinkSSL.java:651) at com.tibco.tibjms.TibjmsxLinkTcp$LinkReader.work(TibjmsxLinkTcp.java:280) at com.tibco.tibjms.TibjmsxLinkTcp$LinkReader.run(TibjmsxLinkTcp.java:259) Locked ownable synchronizers: - None "TIBCO EMS Connections Pinger" - Thread t@90 java.lang.Thread.State: TIMED_WAITING at java.lang.Thread.sleep(Native Method) at com.tibco.tibjms.TibjmsxLinkTcp$Pinger.run(TibjmsxLinkTcp.java:154) at java.lang.Thread.run(Unknown Source) Locked ownable synchronizers: - None "Java2D Disposer" - Thread t@51 java.lang.Thread.State: WAITING on java.lang.ref.ReferenceQueue$Lock@67dfae at java.lang.Object.wait(Native Method) at java.lang.ref.ReferenceQueue.remove(Unknown Source) at java.lang.ref.ReferenceQueue.remove(Unknown Source) at sun.java2d.Disposer.run(Unknown Source) at java.lang.Thread.run(Unknown Source) Locked ownable synchronizers: - None "http-bio-8443-AsyncTimeout" - Thread t@32 java.lang.Thread.State: TIMED_WAITING at java.lang.Thread.sleep(Native Method) at org.apache.tomcat.util.net.JIoEndpoint$AsyncTimeout.run(JIoEndpoint.java:141) at java.lang.Thread.run(Unknown Source) Locked ownable synchronizers: - None "http-bio-8443-Acceptor-0" - Thread t@31 java.lang.Thread.State: WAITING on org.apache.tomcat.util.threads.LimitLatch$Sync@16dd685 at sun.misc.Unsafe.park(Native Method) at java.util.concurrent.locks.LockSupport.park(Unknown Source) at java.util.concurrent.locks.AbstractQueuedSynchronizer.parkAndCheckInterrupt(Unknown Source) at java.util.concurrent.locks.AbstractQueuedSynchronizer.doAcquireSharedInterruptibly(Unknown Source) at java.util.concurrent.locks.AbstractQueuedSynchronizer.acquireSharedInterruptibly(Unknown Source) at org.apache.tomcat.util.threads.LimitLatch.countUpOrAwait(LimitLatch.java:99) at org.apache.tomcat.util.net.AbstractEndpoint.countUpOrAwaitConnection(AbstractEndpoint.java:581) at org.apache.tomcat.util.net.JIoEndpoint$Acceptor.run(JIoEndpoint.java:204) at java.lang.Thread.run(Unknown Source) Locked ownable synchronizers: - None "ContainerBackgroundProcessor[StandardEngine[Catalina]]" - Thread t@30 java.lang.Thread.State: TIMED_WAITING at java.lang.Thread.sleep(Native Method) at org.apache.catalina.core.ContainerBase$ContainerBackgroundProcessor.run(ContainerBase.java:1369) at java.lang.Thread.run(Unknown Source) Locked ownable synchronizers: - None "GC Daemon" - Thread t@21 java.lang.Thread.State: TIMED_WAITING on sun.misc.GC$LatencyLock@18e5957 at java.lang.Object.wait(Native Method) at sun.misc.GC$Daemon.run(Unknown Source) Locked ownable synchronizers: - None "RMI Scheduler(0)" - Thread t@14 java.lang.Thread.State: TIMED_WAITING on java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@186b6cc at sun.misc.Unsafe.park(Native Method) at java.util.concurrent.locks.LockSupport.parkNanos(Unknown Source) at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(Unknown Source) at java.util.concurrent.DelayQueue.take(Unknown Source) at java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(Unknown Source) at java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(Unknown Source) at java.util.concurrent.ThreadPoolExecutor.getTask(Unknown Source) at java.util.concurrent.ThreadPoolExecutor$Worker.run(Unknown Source) at java.lang.Thread.run(Unknown Source) Locked ownable synchronizers: - None "RMI TCP Accept-7360" - Thread t@11 java.lang.Thread.State: RUNNABLE at java.net.PlainSocketImpl.socketAccept(Native Method) at java.net.PlainSocketImpl.accept(Unknown Source) - locked java.net.SocksSocketImpl@1ad9253 at java.net.ServerSocket.implAccept(Unknown Source) at java.net.ServerSocket.accept(Unknown Source) at sun.rmi.transport.tcp.TCPTransport$AcceptLoop.executeAcceptLoop(Unknown Source) at sun.rmi.transport.tcp.TCPTransport$AcceptLoop.run(Unknown Source) at java.lang.Thread.run(Unknown Source) Locked ownable synchronizers: - None "RMI TCP Accept-0" - Thread t@10 java.lang.Thread.State: RUNNABLE at java.net.PlainSocketImpl.socketAccept(Native Method) at java.net.PlainSocketImpl.accept(Unknown Source) - locked java.net.SocksSocketImpl@61c8a1 at java.net.ServerSocket.implAccept(Unknown Source) at java.net.ServerSocket.accept(Unknown Source) at sun.rmi.transport.tcp.TCPTransport$AcceptLoop.executeAcceptLoop(Unknown Source) at sun.rmi.transport.tcp.TCPTransport$AcceptLoop.run(Unknown Source) at java.lang.Thread.run(Unknown Source) Locked ownable synchronizers: - None "Signal Dispatcher" - Thread t@4 java.lang.Thread.State: RUNNABLE Locked ownable synchronizers: - None "Finalizer" - Thread t@3 java.lang.Thread.State: WAITING on java.lang.ref.ReferenceQueue$Lock@14b402d at java.lang.Object.wait(Native Method) at java.lang.ref.ReferenceQueue.remove(Unknown Source) at java.lang.ref.ReferenceQueue.remove(Unknown Source) at java.lang.ref.Finalizer$FinalizerThread.run(Unknown Source) Locked ownable synchronizers: - None "Reference Handler" - Thread t@2 java.lang.Thread.State: WAITING on java.lang.ref.Reference$Lock@16269cb at java.lang.Object.wait(Native Method) at java.lang.Object.wait(Object.java:485) at java.lang.ref.Reference$ReferenceHandler.run(Unknown Source) Locked ownable synchronizers: - None "main" - Thread t@1 java.lang.Thread.State: RUNNABLE at java.net.PlainSocketImpl.socketAccept(Native Method) at java.net.PlainSocketImpl.accept(Unknown Source) - locked java.net.SocksSocketImpl@94237f at java.net.ServerSocket.implAccept(Unknown Source) at java.net.ServerSocket.accept(Unknown Source) at org.apache.catalina.core.StandardServer.await(StandardServer.java:447) at org.apache.catalina.startup.Catalina.await(Catalina.java:709) at org.apache.catalina.startup.Catalina.start(Catalina.java:654) at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) at sun.reflect.NativeMethodAccessorImpl.invoke(Unknown Source) at sun.reflect.DelegatingMethodAccessorImpl.invoke(Unknown Source) at java.lang.reflect.Method.invoke(Unknown Source) at org.apache.catalina.startup.Bootstrap.start(Bootstrap.java:322) at org.apache.catalina.startup.Bootstrap.main(Bootstrap.java:450) Locked ownable synchronizers: - None Also, we are using Default JPA Connection Pooling? Should we use Tomcat JDBC Connection Pooling? There nothing additional configured in context.xml. I see UserDatabaseRealm in server.xml, is this a default setting? What connector is recommended for SSL Connections on Solaris 10? Regards,Bal. Balvindar, On 9/3/12 9:12 AM, balvindar dhaliwal wrote: We are using tomcat 7.0.19, for a jsf(richfaces),seam, jpa web application. Here are the software versions and hardware infrastructure details. Apache Tomact # 7.0.19 Java run time # 1.6.0_12 If possible, it would be best to upgrade to the latest version of both of these packages: Java 1.6.0_36 and Tomcat 7.0.30. There are several important security fixes in both of those (relative to where you are now). We have noticed after about 2 months, the web requests stop being processed. We had observed this issue twice in the last 5 months. We are using HTTP/1.1 Connector ( BIO ). The heap and permgen memory seem fine. There is no out of memory errors in the logs. We have used jvisualvm to monitor heap, permgen, threads and thread dump, AND THEY ALL LOOK OK. The thread dump looks fine and there are no deadlocks/stuck threads observed. Your thread dump attachment was stripped from the list. Can you copy/paste it into a follow post? Please let me know if you need more information. It's hard to guess without the thread dump, but my initial reaction is that it's connection-pool exhaustion, but you said that your connection pool seems healthy. I see from your server.xml that you are using UserDatabaseRealm: is that actually in use in production? It's probably an inappropriate component to use for anything but the most trivial configuration (like maybe the Tomcat manager app). Are you using any kind of <Realm> or <Resource> specifically configured in in your webapp's context.xml? That would also be good to know. If you are using JDBCRealm, you may have hit an edge case that causes some kind of deadlock (but again, you say there are no deadlocks). There were some fixes to the NIO and APR connectors recently (and your version of Tomcat is now more than a year out-of-date) that fixed apparent deadlocks but I'm not sure if those extended into the BIO realm. I can't see anything in the changelog that would indicate a fix to the BIO connector that would affect you. Again, it's very difficult to speculate without some more data, so sending that thread dump (especially if you can give us several thread dumps over a few minutes) will definitely help. > From: balvind...@hotmail.com > To: users@tomcat.apache.org > Subject: RE: Apache tomcat ( 7.0.19 ) stops processing user requests suddenly > but works fine after restart. > Date: Mon, 3 Sep 2012 13:47:50 +0000 > > > > Hi, > Q1:server.xml :------------<?xml version='1.0' encoding='utf-8'?><Server > port="8009" shutdown="xxxxx"> <Listener > className="org.apache.catalina.security.SecurityListener" /> <Listener > className="org.apache.catalina.core.AprLifecycleListener" SSLEngine="on" /> > <Listener className="org.apache.catalina.core.JasperListener" /> <Listener > className="org.apache.catalina.core.JreMemoryLeakPreventionListener" /> > <Listener > className="org.apache.catalina.mbeans.GlobalResourcesLifecycleListener" /> > <Listener > className="org.apache.catalina.core.ThreadLocalLeakPreventionListener" /> > <GlobalNamingResources> <Resource name="UserDatabase" auth="Container" > type="org.apache.catalina.UserDatabase" > description="User database that can be updated and saved" > factory="org.apache.catalina.users.MemoryUserDatabaseFactory" > pathname="conf/tomcat-users.xml" /> </GlobalNamingResources> <Service > name="Catalina"> <Connector port="8443" protocol="HTTP/1.1" > SSLEnabled="true" maxThreads="150" scheme="https" secure="true" > clientAuth="false" sslProtocol="TLS" > ciphers="TLS_RSA_WITH_AES_128_CBC_SHA,TLS_DHE_RSA_WITH_AES_128_CBC_SHA,TLS_DHE_DSS_WITH_AES_128_CBC_SHA,SSL_RSA_WITH_3DES_EDE_CBC_SHA,SSL_DHE_RSA_WITH_3DES_EDE_CBC_SHA,SSL_DHE_DSS_WITH_3DES_EDE_CBC_SHA" > keystoreFile="conf/xxxxxx-yy-zzzz.jks" keystorePass="xxxx" /> > <Engine name="Catalina" defaultHost="localhost"> <Realm > className="org.apache.catalina.realm.UserDatabaseRealm" > digest="SHA" resourceName="UserDatabase"/> <Host name="localhost" > appBase="webapps" unpackWARs="true" autoDeploy="true" > xmlValidation="false" xmlNamespaceAware="false"> > <Valve className="org.apache.catalina.valves.AccessLogValve" > directory="logs" prefix="localhost_access_log." suffix=".txt" > pattern="common" resolveHosts="false"/> > </Host> </Engine> </Service></Server> > Q2:We are using HTTPS. > Regards,Bal. > > Date: Mon, 3 Sep 2012 15:35:31 +0200 > > From: a...@ice-sa.com > > To: users@tomcat.apache.org > > Subject: Re: Apache tomcat ( 7.0.19 ) stops processing user requests > > suddenly but works fine after restart. > > > > balvindar dhaliwal wrote: > > > > > > > > > > > > > > > Hi , > > > > > > > > > > We are using tomcat 7.0.19, for a jsf(richfaces),seam, jpa web > > > application. > > > Here are the software versions and hardware infrastructure details. > > > Apache Tomact # 7.0.19 > > > Java run time # 1.6.0_12 > > > Server infrastructure # SunOS xxxxxxx 5.10 Generic_142900-03 sun4v sparc > > > SUNW,Sun-Fire-T200 > > > OS version # solaris 10. > > > > > > Issue Description # > > > We have noticed after about 2 months, the web requests stop being > > > processed. We had observed this issue twice in the last 5 months. > > > We are using HTTP/1.1 Connector ( BIO ). The heap and permgen memory seem > > > fine. There is no out of memory errors in the logs. We have used > > > jvisualvm to monitor heap, permgen, threads and thread dump, AND THEY ALL > > > LOOK OK. The thread dump looks fine and there are no deadlocks/stuck > > > threads observed. > > > The localhost_access_log stops being updated. The connector port is still > > > listening on SSL port 8443. > > > After we restart Tomcat, the new requests are processing fine. > > > Can you please advise other areas we should look into for investigation > > > (any additional tracing). > > > Should we upgrade to tomcat 7.0.29? What could cause the Connector to > > > stop processing requests on the port? > > > Further info: Heap(1/6th used), PermGen(1/2 used). GC is effectively > > > reducing the heap. The fact that local access log is not being updated, > > > suggests the request is not making to the Servlet container. We are using > > > JPA Connection Pool, have not noticed any db connection issues. After the > > > restart, everything works well. > > > The system logs seems to be fine. > > > Please find the attached sample thread dump copy when this issue was > > > occurred. > > > > > > Please let me know if you need more information. > > > > > > > Q1: Can you copy and paste the server.xml of Tomcat here ? (removing any > > private information). > > Copy and paste in-line, the list strips most attachments. > > > > Q2: Above, you mention that the HTTPS connector on port 8443 is still > > listening. What > > about the HTTP connector on port ??? (usually 80). > > Or are all the requests HTTPS ? > > > > > > > > > > > > --------------------------------------------------------------------- > > To unsubscribe, e-mail: users-unsubscr...@tomcat.apache.org > > For additional commands, e-mail: users-h...@tomcat.apache.org > > > --------------------------------------------------------------------- To unsubscribe, e-mail: users-unsubscr...@tomcat.apache.org For additional commands, e-mail: users-h...@tomcat.apache.org
