Tim,
I am working with Richard on this issue, thanks for your help.
I have verified that the queues can be browsed through the web console
during the pause period.
I pulled a thread dump which is below. I also noticed the following stack
trace on the consuming application (JRuby) when I shut it down for restart.
I'm not sure if it is related or not.
rolling back transaction
(TX:ID:jms03-bbc-prod-34001-1441026436033-1:1:2529913) post failover
recovery. 1 previously delivered message(s) not replayed to consumer:
ID:jms03-bbc-prod-34001-1441026436033-1:1:8:1
org.apache.activemq.ActiveMQMessageConsumer.rollbackOnFailedRecoveryRedelivery(org/apache/activemq/ActiveMQMessageConsumer.java:1173)
org.apache.activemq.ActiveMQMessageConsumer.acknowledge(org/apache/activemq/ActiveMQMessageConsumer.java:1107)
org.apache.activemq.ActiveMQMessageConsumer$5.beforeEnd(org/apache/activemq/ActiveMQMessageConsumer.java:1070)
org.apache.activemq.TransactionContext.beforeEnd(org/apache/activemq/TransactionContext.java:207)
org.apache.activemq.TransactionContext.commit(org/apache/activemq/TransactionContext.java:304)
org.apache.activemq.ActiveMQSession.commit(org/apache/activemq/ActiveMQSession.java:582)
java.lang.reflect.Method.invoke(java/lang/reflect/Method.java:606)
RUBY.consume_queue(/var/www/rails_apps/jms_inbound/releases/20150828210715/app/models/jms_client_instance.rb:141)
org.jruby.RubyProc.call(org/jruby/RubyProc.java:271)
RUBY.session(/home/deploy/rails_apps/jms_inbound/shared/bundle/jruby/1.9/gems/jruby-jms-1.1.0-java/lib/jms/connection.rb:277)
RUBY.consume_queue(/var/www/rails_apps/jms_inbound/releases/20150828210715/app/models/jms_client_instance.rb:132)
java.lang.Thread.run(java/lang/Thread.java:745)
And here is the thread dump from amq during the pause period:
2015-09-02 15:21:53
Full thread dump OpenJDK 64-Bit Server VM (24.79-b02 mixed mode):
"ActiveMQ BrokerService[localhost] Task-13304" daemon prio=10
tid=0x00007fc56c009800 nid=0x7c95 waiting on condition [0x00007fc5a32f1000]
java.lang.Thread.State: TIMED_WAITING (parking)
at sun.misc.Unsafe.park(Native Method)
- parking to wait for <0x00000000c0445b48> (a
java.util.concurrent.SynchronousQueue$TransferStack)
at
java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:226)
at
java.util.concurrent.SynchronousQueue$TransferStack.awaitFulfill(SynchronousQueue.java:460)
at
java.util.concurrent.SynchronousQueue$TransferStack.transfer(SynchronousQueue.java:359)
at java.util.concurrent.SynchronousQueue.poll(SynchronousQueue.java:942)
at
java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1068)
at
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1130)
at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
at java.lang.Thread.run(Thread.java:745)
"RMI TCP Connection(5281)-127.0.0.1" daemon prio=10 tid=0x0000000001e9d800
nid=0x7c94 runnable [0x00007fc5a2cea000]
java.lang.Thread.State: RUNNABLE
at java.net.SocketInputStream.socketRead0(Native Method)
at java.net.SocketInputStream.read(SocketInputStream.java:152)
at java.net.SocketInputStream.read(SocketInputStream.java:122)
at java.io.BufferedInputStream.fill(BufferedInputStream.java:235)
at java.io.BufferedInputStream.read(BufferedInputStream.java:254)
- locked <0x00000000fc7a74e8> (a java.io.BufferedInputStream)
at java.io.FilterInputStream.read(FilterInputStream.java:83)
at
sun.rmi.transport.tcp.TCPTransport.handleMessages(TCPTransport.java:549)
at
sun.rmi.transport.tcp.TCPTransport$ConnectionHandler.run0(TCPTransport.java:828)
at
sun.rmi.transport.tcp.TCPTransport$ConnectionHandler.access$400(TCPTransport.java:619)
at
sun.rmi.transport.tcp.TCPTransport$ConnectionHandler$1.run(TCPTransport.java:684)
at
sun.rmi.transport.tcp.TCPTransport$ConnectionHandler$1.run(TCPTransport.java:681)
at java.security.AccessController.doPrivileged(Native Method)
at
sun.rmi.transport.tcp.TCPTransport$ConnectionHandler.run(TCPTransport.java:681)
at
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
at java.lang.Thread.run(Thread.java:745)
"Attach Listener" daemon prio=10 tid=0x00007fc584001000 nid=0x7c93 waiting
on condition [0x0000000000000000]
java.lang.Thread.State: RUNNABLE
"ActiveMQ InactivityMonitor Worker" daemon prio=10 tid=0x00007fc534017000
nid=0x7c64 waiting on condition [0x00007fc5a3dfc000]
java.lang.Thread.State: TIMED_WAITING (parking)
at sun.misc.Unsafe.park(Native Method)
- parking to wait for <0x00000000c2903e48> (a
java.util.concurrent.SynchronousQueue$TransferStack)
at
java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:226)
at
java.util.concurrent.SynchronousQueue$TransferStack.awaitFulfill(SynchronousQueue.java:460)
at
java.util.concurrent.SynchronousQueue$TransferStack.transfer(SynchronousQueue.java:359)
at java.util.concurrent.SynchronousQueue.poll(SynchronousQueue.java:942)
at
java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1068)
at
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1130)
at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
at java.lang.Thread.run(Thread.java:745)
"JMX server connection timeout 42176" daemon prio=10 tid=0x00007fc5a40e3000
nid=0x721f in Object.wait() [0x00007fc5a33f2000]
java.lang.Thread.State: TIMED_WAITING (on object monitor)
at java.lang.Object.wait(Native Method)
at
com.sun.jmx.remote.internal.ServerCommunicatorAdmin$Timeout.run(ServerCommunicatorAdmin.java:168)
- locked <0x00000000fa3d9f28> (a [I)
at java.lang.Thread.run(Thread.java:745)
"JMX server connection timeout 42175" daemon prio=10 tid=0x00007fc5a404b800
nid=0x7208 in Object.wait() [0x00007fc5a29e8000]
java.lang.Thread.State: TIMED_WAITING (on object monitor)
at java.lang.Object.wait(Native Method)
at
com.sun.jmx.remote.internal.ServerCommunicatorAdmin$Timeout.run(ServerCommunicatorAdmin.java:168)
- locked <0x00000000fa3bea20> (a [I)
at java.lang.Thread.run(Thread.java:745)
"ActiveMQ Transport: tcp:///173.247.107.193:40101@61616" daemon prio=10
tid=0x00007fc5a4082000 nid=0x71ed runnable [0x00007fc5a97a0000]
java.lang.Thread.State: RUNNABLE
at java.net.SocketInputStream.socketRead0(Native Method)
at java.net.SocketInputStream.read(SocketInputStream.java:152)
at java.net.SocketInputStream.read(SocketInputStream.java:122)
at
org.apache.activemq.transport.tcp.TcpBufferedInputStream.fill(TcpBufferedInputStream.java:50)
at
org.apache.activemq.transport.tcp.TcpTransport$2.fill(TcpTransport.java:609)
at
org.apache.activemq.transport.tcp.TcpBufferedInputStream.read(TcpBufferedInputStream.java:58)
at
org.apache.activemq.transport.tcp.TcpTransport$2.read(TcpTransport.java:594)
at java.io.DataInputStream.readInt(DataInputStream.java:387)
at
org.apache.activemq.openwire.OpenWireFormat.unmarshal(OpenWireFormat.java:258)
at
org.apache.activemq.transport.tcp.TcpTransport.readCommand(TcpTransport.java:221)
at
org.apache.activemq.transport.tcp.TcpTransport.doRun(TcpTransport.java:213)
at
org.apache.activemq.transport.tcp.TcpTransport.run(TcpTransport.java:196)
at java.lang.Thread.run(Thread.java:745)
"ActiveMQ BrokerService[localhost] Task-13300" daemon prio=10
tid=0x00007fc5a40e5000 nid=0x715f waiting on condition [0x00007fc5a30ef000]
java.lang.Thread.State: TIMED_WAITING (parking)
at sun.misc.Unsafe.park(Native Method)
- parking to wait for <0x00000000c0445b48> (a
java.util.concurrent.SynchronousQueue$TransferStack)
at
java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:226)
at
java.util.concurrent.SynchronousQueue$TransferStack.awaitFulfill(SynchronousQueue.java:460)
at
java.util.concurrent.SynchronousQueue$TransferStack.transfer(SynchronousQueue.java:359)
at java.util.concurrent.SynchronousQueue.poll(SynchronousQueue.java:942)
at
java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1068)
at
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1130)
at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
at java.lang.Thread.run(Thread.java:745)
"ActiveMQ BrokerService.worker.1" daemon prio=10 tid=0x00007fc54801e000
nid=0x1c79 waiting on condition [0x00007fc5a38f7000]
java.lang.Thread.State: WAITING (parking)
at sun.misc.Unsafe.park(Native Method)
- parking to wait for <0x00000000c055d4a0> (a
java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
at java.util.concurrent.locks.LockSupport.park(LockSupport.java:186)
at
java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2043)
at
java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:442)
at
java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1068)
at
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1130)
at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
at java.lang.Thread.run(Thread.java:745)
"ActiveMQ Transport: tcp:///173.247.107.206:51688@61616" daemon prio=10
tid=0x0000000001e36000 nid=0x1a6a runnable [0x00007fc5a36f5000]
java.lang.Thread.State: RUNNABLE
at java.net.SocketInputStream.socketRead0(Native Method)
at java.net.SocketInputStream.read(SocketInputStream.java:152)
at java.net.SocketInputStream.read(SocketInputStream.java:122)
at
org.apache.activemq.transport.tcp.TcpBufferedInputStream.fill(TcpBufferedInputStream.java:50)
at
org.apache.activemq.transport.tcp.TcpTransport$2.fill(TcpTransport.java:609)
at
org.apache.activemq.transport.tcp.TcpBufferedInputStream.read(TcpBufferedInputStream.java:58)
at
org.apache.activemq.transport.tcp.TcpTransport$2.read(TcpTransport.java:594)
at java.io.DataInputStream.readInt(DataInputStream.java:387)
at
org.apache.activemq.openwire.OpenWireFormat.unmarshal(OpenWireFormat.java:258)
at
org.apache.activemq.transport.tcp.TcpTransport.readCommand(TcpTransport.java:221)
at
org.apache.activemq.transport.tcp.TcpTransport.doRun(TcpTransport.java:213)
at
org.apache.activemq.transport.tcp.TcpTransport.run(TcpTransport.java:196)
at java.lang.Thread.run(Thread.java:745)
"ActiveMQ Transport: tcp:///173.247.107.206:51687@61616" daemon prio=10
tid=0x00007fc554043800 nid=0x1a69 runnable [0x00007fc5a37f6000]
java.lang.Thread.State: RUNNABLE
at java.net.SocketInputStream.socketRead0(Native Method)
at java.net.SocketInputStream.read(SocketInputStream.java:152)
at java.net.SocketInputStream.read(SocketInputStream.java:122)
at
org.apache.activemq.transport.tcp.TcpBufferedInputStream.fill(TcpBufferedInputStream.java:50)
at
org.apache.activemq.transport.tcp.TcpTransport$2.fill(TcpTransport.java:609)
at
org.apache.activemq.transport.tcp.TcpBufferedInputStream.read(TcpBufferedInputStream.java:58)
at
org.apache.activemq.transport.tcp.TcpTransport$2.read(TcpTransport.java:594)
at java.io.DataInputStream.readInt(DataInputStream.java:387)
at
org.apache.activemq.openwire.OpenWireFormat.unmarshal(OpenWireFormat.java:258)
at
org.apache.activemq.transport.tcp.TcpTransport.readCommand(TcpTransport.java:221)
at
org.apache.activemq.transport.tcp.TcpTransport.doRun(TcpTransport.java:213)
at
org.apache.activemq.transport.tcp.TcpTransport.run(TcpTransport.java:196)
at java.lang.Thread.run(Thread.java:745)
"ActiveMQ InactivityMonitor WriteCheckTimer" daemon prio=10
tid=0x00007fc59c008800 nid=0x1a64 in Object.wait() [0x00007fc5a3bfa000]
java.lang.Thread.State: TIMED_WAITING (on object monitor)
at java.lang.Object.wait(Native Method)
at java.util.TimerThread.mainLoop(Timer.java:552)
- locked <0x00000000c2903568> (a java.util.TaskQueue)
at java.util.TimerThread.run(Timer.java:505)
"ActiveMQ InactivityMonitor ReadCheckTimer" daemon prio=10
tid=0x00007fc59c020000 nid=0x1a63 in Object.wait() [0x00007fc5a39f8000]
java.lang.Thread.State: TIMED_WAITING (on object monitor)
at java.lang.Object.wait(Native Method)
at java.util.TimerThread.mainLoop(Timer.java:552)
- locked <0x00000000c29039a8> (a java.util.TaskQueue)
at java.util.TimerThread.run(Timer.java:505)
"qtp1455892368-90" prio=10 tid=0x00007fc5b9073000 nid=0x58a waiting on
condition [0x00007fc5a3efd000]
java.lang.Thread.State: TIMED_WAITING (parking)
at sun.misc.Unsafe.park(Native Method)
- parking to wait for <0x00000000c04687f8> (a
java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
at
java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:226)
at
java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2082)
at
org.eclipse.jetty.util.BlockingArrayQueue.poll(BlockingArrayQueue.java:342)
at
org.eclipse.jetty.util.thread.QueuedThreadPool.idleJobPoll(QueuedThreadPool.java:526)
at
org.eclipse.jetty.util.thread.QueuedThreadPool.access$600(QueuedThreadPool.java:44)
at
org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:572)
at java.lang.Thread.run(Thread.java:745)
"qtp1455892368-89" prio=10 tid=0x00007fc5b9071000 nid=0x589 waiting on
condition [0x00007fc5a3ffe000]
java.lang.Thread.State: TIMED_WAITING (parking)
at sun.misc.Unsafe.park(Native Method)
- parking to wait for <0x00000000c04687f8> (a
java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
at
java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:226)
at
java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2082)
at
org.eclipse.jetty.util.BlockingArrayQueue.poll(BlockingArrayQueue.java:342)
at
org.eclipse.jetty.util.thread.QueuedThreadPool.idleJobPoll(QueuedThreadPool.java:526)
at
org.eclipse.jetty.util.thread.QueuedThreadPool.access$600(QueuedThreadPool.java:44)
at
org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:572)
at java.lang.Thread.run(Thread.java:745)
"qtp1455892368-88" prio=10 tid=0x00007fc5b906f000 nid=0x588 waiting on
condition [0x00007fc5a818a000]
java.lang.Thread.State: TIMED_WAITING (parking)
at sun.misc.Unsafe.park(Native Method)
- parking to wait for <0x00000000c04687f8> (a
java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
at
java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:226)
at
java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2082)
at
org.eclipse.jetty.util.BlockingArrayQueue.poll(BlockingArrayQueue.java:342)
at
org.eclipse.jetty.util.thread.QueuedThreadPool.idleJobPoll(QueuedThreadPool.java:526)
at
org.eclipse.jetty.util.thread.QueuedThreadPool.access$600(QueuedThreadPool.java:44)
at
org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:572)
at java.lang.Thread.run(Thread.java:745)
"qtp1455892368-87" prio=10 tid=0x00007fc5b906d000 nid=0x587 waiting on
condition [0x00007fc5a828b000]
java.lang.Thread.State: TIMED_WAITING (parking)
at sun.misc.Unsafe.park(Native Method)
- parking to wait for <0x00000000c04687f8> (a
java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
at
java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:226)
at
java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2082)
at
org.eclipse.jetty.util.BlockingArrayQueue.poll(BlockingArrayQueue.java:342)
at
org.eclipse.jetty.util.thread.QueuedThreadPool.idleJobPoll(QueuedThreadPool.java:526)
at
org.eclipse.jetty.util.thread.QueuedThreadPool.access$600(QueuedThreadPool.java:44)
at
org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:572)
at java.lang.Thread.run(Thread.java:745)
"qtp1455892368-86" prio=10 tid=0x00007fc5b906a800 nid=0x586 waiting on
condition [0x00007fc5a838c000]
java.lang.Thread.State: TIMED_WAITING (parking)
at sun.misc.Unsafe.park(Native Method)
- parking to wait for <0x00000000c04687f8> (a
java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
at
java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:226)
at
java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2082)
at
org.eclipse.jetty.util.BlockingArrayQueue.poll(BlockingArrayQueue.java:342)
at
org.eclipse.jetty.util.thread.QueuedThreadPool.idleJobPoll(QueuedThreadPool.java:526)
at
org.eclipse.jetty.util.thread.QueuedThreadPool.access$600(QueuedThreadPool.java:44)
at
org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:572)
at java.lang.Thread.run(Thread.java:745)
"qtp1455892368-85" prio=10 tid=0x00007fc5b9069000 nid=0x585 waiting on
condition [0x00007fc5a848d000]
java.lang.Thread.State: TIMED_WAITING (parking)
at sun.misc.Unsafe.park(Native Method)
- parking to wait for <0x00000000c04687f8> (a
java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
at
java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:226)
at
java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2082)
at
org.eclipse.jetty.util.BlockingArrayQueue.poll(BlockingArrayQueue.java:342)
at
org.eclipse.jetty.util.thread.QueuedThreadPool.idleJobPoll(QueuedThreadPool.java:526)
at
org.eclipse.jetty.util.thread.QueuedThreadPool.access$600(QueuedThreadPool.java:44)
at
org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:572)
at java.lang.Thread.run(Thread.java:745)
"qtp1455892368-84 Acceptor0 SelectChannelConnector@0.0.0.0:61614" prio=10
tid=0x00007fc5b905b800 nid=0x584 runnable [0x00007fc5a858e000]
java.lang.Thread.State: RUNNABLE
at sun.nio.ch.ServerSocketChannelImpl.accept0(Native Method)
at
sun.nio.ch.ServerSocketChannelImpl.accept(ServerSocketChannelImpl.java:241)
- locked <0x00000000c0468468> (a java.lang.Object)
at
org.eclipse.jetty.server.nio.SelectChannelConnector.accept(SelectChannelConnector.java:109)
at
org.eclipse.jetty.server.AbstractConnector$Acceptor.run(AbstractConnector.java:938)
at
org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:608)
at
org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:543)
at java.lang.Thread.run(Thread.java:745)
"qtp1455892368-83 Selector0" prio=10 tid=0x00007fc5b905d000 nid=0x583
runnable [0x00007fc5a868f000]
java.lang.Thread.State: RUNNABLE
at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:269)
at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:79)
at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:87)
- locked <0x00000000c0466f18> (a sun.nio.ch.Util$2)
- locked <0x00000000c0466f08> (a java.util.Collections$UnmodifiableSet)
- locked <0x00000000c0466dd0> (a sun.nio.ch.EPollSelectorImpl)
at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:98)
at
org.eclipse.jetty.io.nio.SelectorManager$SelectSet.doSelect(SelectorManager.java:569)
at
org.eclipse.jetty.io.nio.SelectorManager$1.run(SelectorManager.java:290)
at
org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:608)
at
org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:543)
at java.lang.Thread.run(Thread.java:745)
"ActiveMQ Transport Server:
mqtt://0.0.0.0:1883?maximumConnections=1000&wireFormat.maxFrameSize=104857600"
daemon prio=10 tid=0x00007fc5b84af000 nid=0x582 runnable
[0x00007fc5a8790000]
java.lang.Thread.State: RUNNABLE
at java.net.PlainSocketImpl.socketAccept(Native Method)
at
java.net.AbstractPlainSocketImpl.accept(AbstractPlainSocketImpl.java:398)
at java.net.ServerSocket.implAccept(ServerSocket.java:530)
at java.net.ServerSocket.accept(ServerSocket.java:498)
at
org.apache.activemq.transport.tcp.TcpTransportServer.run(TcpTransportServer.java:350)
at java.lang.Thread.run(Thread.java:745)
"ActiveMQ Transport Server Thread Handler:
mqtt://0.0.0.0:1883?maximumConnections=1000&wireFormat.maxFrameSize=104857600"
daemon prio=10 tid=0x00007fc5b8c85800 nid=0x581 waiting on condition
[0x00007fc5a8891000]
java.lang.Thread.State: TIMED_WAITING (parking)
at sun.misc.Unsafe.park(Native Method)
- parking to wait for <0x00000000c04dd038> (a
java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
at
java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:226)
at
java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2082)
at
java.util.concurrent.LinkedBlockingQueue.poll(LinkedBlockingQueue.java:467)
at
org.apache.activemq.transport.tcp.TcpTransportServer$2.run(TcpTransportServer.java:425)
at java.lang.Thread.run(Thread.java:745)
"ActiveMQ Transport Server:
stomp://0.0.0.0:61613?maximumConnections=1000&wireFormat.maxFrameSize=104857600&transport.defaultHeartBeat=30000,0"
daemon prio=10 tid=0x00007fc5b84a7800 nid=0x580 runnable
[0x00007fc5a8992000]
java.lang.Thread.State: RUNNABLE
at java.net.PlainSocketImpl.socketAccept(Native Method)
at
java.net.AbstractPlainSocketImpl.accept(AbstractPlainSocketImpl.java:398)
at java.net.ServerSocket.implAccept(ServerSocket.java:530)
at java.net.ServerSocket.accept(ServerSocket.java:498)
at
org.apache.activemq.transport.tcp.TcpTransportServer.run(TcpTransportServer.java:350)
at java.lang.Thread.run(Thread.java:745)
"ActiveMQ Transport Server Thread Handler:
stomp://0.0.0.0:61613?maximumConnections=1000&wireFormat.maxFrameSize=104857600&transport.defaultHeartBeat=30000,0"
daemon prio=10 tid=0x00007fc5b8c95800 nid=0x57f waiting on condition
[0x00007fc5a8a93000]
java.lang.Thread.State: TIMED_WAITING (parking)
at sun.misc.Unsafe.park(Native Method)
- parking to wait for <0x00000000c0583740> (a
java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
at
java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:226)
at
java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2082)
at
java.util.concurrent.LinkedBlockingQueue.poll(LinkedBlockingQueue.java:467)
at
org.apache.activemq.transport.tcp.TcpTransportServer$2.run(TcpTransportServer.java:425)
at java.lang.Thread.run(Thread.java:745)
"ActiveMQ Transport Server:
amqp://0.0.0.0:5672?maximumConnections=1000&wireFormat.maxFrameSize=104857600"
daemon prio=10 tid=0x00007fc5b851a800 nid=0x57e runnable
[0x00007fc5a8b94000]
java.lang.Thread.State: RUNNABLE
at java.net.PlainSocketImpl.socketAccept(Native Method)
at
java.net.AbstractPlainSocketImpl.accept(AbstractPlainSocketImpl.java:398)
at java.net.ServerSocket.implAccept(ServerSocket.java:530)
at java.net.ServerSocket.accept(ServerSocket.java:498)
at
org.apache.activemq.transport.tcp.TcpTransportServer.run(TcpTransportServer.java:350)
at java.lang.Thread.run(Thread.java:745)
"ActiveMQ Transport Server Thread Handler:
amqp://0.0.0.0:5672?maximumConnections=1000&wireFormat.maxFrameSize=104857600"
daemon prio=10 tid=0x00007fc5b8a3f000 nid=0x57d waiting on condition
[0x00007fc5a8c95000]
java.lang.Thread.State: TIMED_WAITING (parking)
at sun.misc.Unsafe.park(Native Method)
- parking to wait for <0x00000000c05848b0> (a
java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
at
java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:226)
at
java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2082)
at
java.util.concurrent.LinkedBlockingQueue.poll(LinkedBlockingQueue.java:467)
at
org.apache.activemq.transport.tcp.TcpTransportServer$2.run(TcpTransportServer.java:425)
at java.lang.Thread.run(Thread.java:745)
"ActiveMQ Transport Server:
tcp://0.0.0.0:61616?maximumConnections=1000&wireFormat.maxFrameSize=104857600&transport.trace=true"
daemon prio=10 tid=0x00007fc5b8a47800 nid=0x57c runnable
[0x00007fc5a8d96000]
java.lang.Thread.State: RUNNABLE
at java.net.PlainSocketImpl.socketAccept(Native Method)
at
java.net.AbstractPlainSocketImpl.accept(AbstractPlainSocketImpl.java:398)
at java.net.ServerSocket.implAccept(ServerSocket.java:530)
at java.net.ServerSocket.accept(ServerSocket.java:498)
at
org.apache.activemq.transport.tcp.TcpTransportServer.run(TcpTransportServer.java:350)
at java.lang.Thread.run(Thread.java:745)
"ActiveMQ Transport Server Thread Handler:
tcp://0.0.0.0:61616?maximumConnections=1000&wireFormat.maxFrameSize=104857600&transport.trace=true"
daemon prio=10 tid=0x00007fc5b8a46800 nid=0x57b waiting on condition
[0x00007fc5a8e97000]
java.lang.Thread.State: TIMED_WAITING (parking)
at sun.misc.Unsafe.park(Native Method)
- parking to wait for <0x00000000c0585e18> (a
java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
at
java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:226)
at
java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2082)
at
java.util.concurrent.LinkedBlockingQueue.poll(LinkedBlockingQueue.java:467)
at
org.apache.activemq.transport.tcp.TcpTransportServer$2.run(TcpTransportServer.java:425)
at java.lang.Thread.run(Thread.java:745)
"ConcurrentQueueStoreAndDispatch" daemon prio=10 tid=0x00007fc54c004800
nid=0x579 waiting on condition [0x00007fc5a9099000]
java.lang.Thread.State: WAITING (parking)
at sun.misc.Unsafe.park(Native Method)
- parking to wait for <0x00000000c055c0b8> (a
java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
at java.util.concurrent.locks.LockSupport.park(LockSupport.java:186)
at
java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2043)
at
java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:442)
at
java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1068)
at
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1130)
at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
at java.lang.Thread.run(Thread.java:745)
"JobScheduler:JMS" daemon prio=10 tid=0x00007fc5b8a48800 nid=0x578 in
Object.wait() [0x00007fc5a919a000]
java.lang.Thread.State: TIMED_WAITING (on object monitor)
at java.lang.Object.wait(Native Method)
at
org.apache.activemq.store.kahadb.scheduler.JobSchedulerImpl$ScheduleTime.pause(JobSchedulerImpl.java:895)
- locked <0x00000000c05865b8> (a java.lang.Object)
at
org.apache.activemq.store.kahadb.scheduler.JobSchedulerImpl.mainLoop(JobSchedulerImpl.java:775)
at
org.apache.activemq.store.kahadb.scheduler.JobSchedulerImpl.run(JobSchedulerImpl.java:673)
at java.lang.Thread.run(Thread.java:745)
"ActiveMQ Data File Writer" daemon prio=10 tid=0x00007fc5b8b4d800 nid=0x577
in Object.wait() [0x00007fc5a929b000]
java.lang.Thread.State: WAITING (on object monitor)
at java.lang.Object.wait(Native Method)
at java.lang.Object.wait(Object.java:503)
at
org.apache.activemq.store.kahadb.disk.journal.DataFileAppender.processQueue(DataFileAppender.java:304)
- locked <0x00000000c053a8e0> (a java.lang.Object)
at
org.apache.activemq.store.kahadb.disk.journal.DataFileAppender$1.run(DataFileAppender.java:193)
"ActiveMQ Journal Checkpoint Worker" daemon prio=10 tid=0x00007fc5b8740800
nid=0x576 waiting on condition [0x00007fc5a939c000]
java.lang.Thread.State: TIMED_WAITING (sleeping)
at java.lang.Thread.sleep(Native Method)
at
org.apache.activemq.store.kahadb.MessageDatabase$3.run(MessageDatabase.java:349)
"KahaDB Scheduler" daemon prio=10 tid=0x00007fc5b8c70000 nid=0x575 in
Object.wait() [0x00007fc5a9ba4000]
java.lang.Thread.State: TIMED_WAITING (on object monitor)
at java.lang.Object.wait(Native Method)
at java.util.TimerThread.mainLoop(Timer.java:552)
- locked <0x00000000c055b428> (a java.util.TaskQueue)
at java.util.TimerThread.run(Timer.java:505)
"RMI RenewClean-[127.0.0.1:48814]" daemon prio=10 tid=0x00007fc534005800
nid=0x573 in Object.wait() [0x00007fc5a959e000]
java.lang.Thread.State: TIMED_WAITING (on object monitor)
at java.lang.Object.wait(Native Method)
at java.lang.ref.ReferenceQueue.remove(ReferenceQueue.java:135)
- locked <0x00000000c04432f0> (a java.lang.ref.ReferenceQueue$Lock)
at
sun.rmi.transport.DGCClient$EndpointEntry$RenewCleanThread.run(DGCClient.java:535)
at java.lang.Thread.run(Thread.java:745)
"RMI Scheduler(0)" daemon prio=10 tid=0x00007fc52c00d000 nid=0x572 waiting
on condition [0x00007fc5a969f000]
java.lang.Thread.State: TIMED_WAITING (parking)
at sun.misc.Unsafe.park(Native Method)
- parking to wait for <0x00000000c04449d8> (a
java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
at
java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:226)
at
java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2082)
at
java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:1090)
at
java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:807)
at
java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1068)
at
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1130)
at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
at java.lang.Thread.run(Thread.java:745)
"GC Daemon" daemon prio=10 tid=0x00007fc52c005000 nid=0x570 in Object.wait()
[0x00007fc5a98a1000]
java.lang.Thread.State: TIMED_WAITING (on object monitor)
at java.lang.Object.wait(Native Method)
at sun.misc.GC$Daemon.run(GC.java:117)
- locked <0x00000000c0bf1ca0> (a sun.misc.GC$LatencyLock)
"RMI Reaper" prio=10 tid=0x00007fc52c003000 nid=0x56f in Object.wait()
[0x00007fc5a99a2000]
java.lang.Thread.State: WAITING (on object monitor)
at java.lang.Object.wait(Native Method)
- waiting on <0x00000000c04438d0> (a java.lang.ref.ReferenceQueue$Lock)
at java.lang.ref.ReferenceQueue.remove(ReferenceQueue.java:135)
- locked <0x00000000c04438d0> (a java.lang.ref.ReferenceQueue$Lock)
at java.lang.ref.ReferenceQueue.remove(ReferenceQueue.java:151)
at sun.rmi.transport.ObjectTable$Reaper.run(ObjectTable.java:351)
at java.lang.Thread.run(Thread.java:745)
"RMI TCP Accept-0" daemon prio=10 tid=0x00007fc52c001000 nid=0x56e runnable
[0x00007fc5a9aa3000]
java.lang.Thread.State: RUNNABLE
at java.net.PlainSocketImpl.socketAccept(Native Method)
at
java.net.AbstractPlainSocketImpl.accept(AbstractPlainSocketImpl.java:398)
at java.net.ServerSocket.implAccept(ServerSocket.java:530)
at java.net.ServerSocket.accept(ServerSocket.java:498)
at
sun.rmi.transport.tcp.TCPTransport$AcceptLoop.executeAcceptLoop(TCPTransport.java:399)
at
sun.rmi.transport.tcp.TCPTransport$AcceptLoop.run(TCPTransport.java:371)
at java.lang.Thread.run(Thread.java:745)
"RMI TCP Accept-1099" daemon prio=10 tid=0x00007fc5b8c61000 nid=0x56c
runnable [0x00007fc5a9ca5000]
java.lang.Thread.State: RUNNABLE
at java.net.PlainSocketImpl.socketAccept(Native Method)
at
java.net.AbstractPlainSocketImpl.accept(AbstractPlainSocketImpl.java:398)
at java.net.ServerSocket.implAccept(ServerSocket.java:530)
at java.net.ServerSocket.accept(ServerSocket.java:498)
at
sun.rmi.transport.tcp.TCPTransport$AcceptLoop.executeAcceptLoop(TCPTransport.java:399)
at
sun.rmi.transport.tcp.TCPTransport$AcceptLoop.run(TCPTransport.java:371)
at java.lang.Thread.run(Thread.java:745)
"qtp205494465-25" prio=10 tid=0x00007fc5b8c59000 nid=0x56b waiting on
condition [0x00007fc5a9da6000]
java.lang.Thread.State: TIMED_WAITING (parking)
at sun.misc.Unsafe.park(Native Method)
- parking to wait for <0x00000000c0bf1d18> (a
java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
at
java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:226)
at
java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2082)
at
org.eclipse.jetty.util.BlockingArrayQueue.poll(BlockingArrayQueue.java:342)
at
org.eclipse.jetty.util.thread.QueuedThreadPool.idleJobPoll(QueuedThreadPool.java:526)
at
org.eclipse.jetty.util.thread.QueuedThreadPool.access$600(QueuedThreadPool.java:44)
at
org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:572)
at java.lang.Thread.run(Thread.java:745)
"qtp205494465-24" prio=10 tid=0x00007fc5b8c57000 nid=0x56a waiting on
condition [0x00007fc5a9ea7000]
java.lang.Thread.State: TIMED_WAITING (parking)
at sun.misc.Unsafe.park(Native Method)
- parking to wait for <0x00000000c0bf1d18> (a
java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
at
java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:226)
at
java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2082)
at
org.eclipse.jetty.util.BlockingArrayQueue.poll(BlockingArrayQueue.java:342)
at
org.eclipse.jetty.util.thread.QueuedThreadPool.idleJobPoll(QueuedThreadPool.java:526)
at
org.eclipse.jetty.util.thread.QueuedThreadPool.access$600(QueuedThreadPool.java:44)
at
org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:572)
at java.lang.Thread.run(Thread.java:745)
"qtp205494465-23" prio=10 tid=0x00007fc5b8c55000 nid=0x569 waiting on
condition [0x00007fc5a9fa8000]
java.lang.Thread.State: TIMED_WAITING (parking)
at sun.misc.Unsafe.park(Native Method)
- parking to wait for <0x00000000c0bf1d18> (a
java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
at
java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:226)
at
java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2082)
at
org.eclipse.jetty.util.BlockingArrayQueue.poll(BlockingArrayQueue.java:342)
at
org.eclipse.jetty.util.thread.QueuedThreadPool.idleJobPoll(QueuedThreadPool.java:526)
at
org.eclipse.jetty.util.thread.QueuedThreadPool.access$600(QueuedThreadPool.java:44)
at
org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:572)
at java.lang.Thread.run(Thread.java:745)
"qtp205494465-22" prio=10 tid=0x00007fc5b8c53000 nid=0x568 waiting on
condition [0x00007fc5aa0a9000]
java.lang.Thread.State: TIMED_WAITING (parking)
at sun.misc.Unsafe.park(Native Method)
- parking to wait for <0x00000000c0bf1d18> (a
java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
at
java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:226)
at
java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2082)
at
org.eclipse.jetty.util.BlockingArrayQueue.poll(BlockingArrayQueue.java:342)
at
org.eclipse.jetty.util.thread.QueuedThreadPool.idleJobPoll(QueuedThreadPool.java:526)
at
org.eclipse.jetty.util.thread.QueuedThreadPool.access$600(QueuedThreadPool.java:44)
at
org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:572)
at java.lang.Thread.run(Thread.java:745)
"qtp205494465-21" prio=10 tid=0x00007fc5b8c51800 nid=0x567 waiting on
condition [0x00007fc5aa1aa000]
java.lang.Thread.State: TIMED_WAITING (parking)
at sun.misc.Unsafe.park(Native Method)
- parking to wait for <0x00000000c0bf1d18> (a
java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
at
java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:226)
at
java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2082)
at
org.eclipse.jetty.util.BlockingArrayQueue.poll(BlockingArrayQueue.java:342)
at
org.eclipse.jetty.util.thread.QueuedThreadPool.idleJobPoll(QueuedThreadPool.java:526)
at
org.eclipse.jetty.util.thread.QueuedThreadPool.access$600(QueuedThreadPool.java:44)
at
org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:572)
at java.lang.Thread.run(Thread.java:745)
"qtp205494465-20" prio=10 tid=0x00007fc5b8c47000 nid=0x566 waiting on
condition [0x00007fc5aa2ab000]
java.lang.Thread.State: TIMED_WAITING (parking)
at sun.misc.Unsafe.park(Native Method)
- parking to wait for <0x00000000c0bf1d18> (a
java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
at
java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:226)
at
java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2082)
at
org.eclipse.jetty.util.BlockingArrayQueue.poll(BlockingArrayQueue.java:342)
at
org.eclipse.jetty.util.thread.QueuedThreadPool.idleJobPoll(QueuedThreadPool.java:526)
at
org.eclipse.jetty.util.thread.QueuedThreadPool.access$600(QueuedThreadPool.java:44)
at
org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:572)
at java.lang.Thread.run(Thread.java:745)
"qtp205494465-19 Acceptor0 SelectChannelConnector@0.0.0.0:8161" prio=10
tid=0x00007fc5b8c46000 nid=0x565 runnable [0x00007fc5aa3ac000]
java.lang.Thread.State: RUNNABLE
at sun.nio.ch.ServerSocketChannelImpl.accept0(Native Method)
at
sun.nio.ch.ServerSocketChannelImpl.accept(ServerSocketChannelImpl.java:241)
- locked <0x00000000c0bf27c8> (a java.lang.Object)
at
org.eclipse.jetty.server.nio.SelectChannelConnector.accept(SelectChannelConnector.java:109)
at
org.eclipse.jetty.server.AbstractConnector$Acceptor.run(AbstractConnector.java:938)
at
org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:608)
at
org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:543)
at java.lang.Thread.run(Thread.java:745)
"qtp205494465-18 Selector0" prio=10 tid=0x00007fc5b8c4e800 nid=0x564
runnable [0x00007fc5aa4ad000]
java.lang.Thread.State: RUNNABLE
at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:269)
at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:79)
at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:87)
- locked <0x00000000c0bf2ce0> (a sun.nio.ch.Util$2)
- locked <0x00000000c0bf2cd0> (a java.util.Collections$UnmodifiableSet)
- locked <0x00000000c0bf2b98> (a sun.nio.ch.EPollSelectorImpl)
at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:98)
at
org.eclipse.jetty.io.nio.SelectorManager$SelectSet.doSelect(SelectorManager.java:569)
at
org.eclipse.jetty.io.nio.SelectorManager$1.run(SelectorManager.java:290)
at
org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:608)
at
org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:543)
at java.lang.Thread.run(Thread.java:745)
"HashSessionScavenger-2" daemon prio=10 tid=0x00007fc5b8bee000 nid=0x563 in
Object.wait() [0x00007fc5aa5ae000]
java.lang.Thread.State: TIMED_WAITING (on object monitor)
at java.lang.Object.wait(Native Method)
at java.util.TimerThread.mainLoop(Timer.java:552)
- locked <0x00000000c050fb00> (a java.util.TaskQueue)
at java.util.TimerThread.run(Timer.java:505)
"HashSessionScavenger-1" daemon prio=10 tid=0x00007fc5b8bd7800 nid=0x562 in
Object.wait() [0x00007fc5aa6af000]
java.lang.Thread.State: TIMED_WAITING (on object monitor)
at java.lang.Object.wait(Native Method)
at java.util.TimerThread.mainLoop(Timer.java:552)
- locked <0x00000000c09ee3e8> (a java.util.TaskQueue)
at java.util.TimerThread.run(Timer.java:505)
"HashSessionScavenger-0" daemon prio=10 tid=0x00007fc5b8ae2000 nid=0x561 in
Object.wait() [0x00007fc5aa7ba000]
java.lang.Thread.State: TIMED_WAITING (on object monitor)
at java.lang.Object.wait(Native Method)
at java.util.TimerThread.mainLoop(Timer.java:552)
- locked <0x00000000c0aeeba0> (a java.util.TaskQueue)
at java.util.TimerThread.run(Timer.java:505)
"ActiveMQ Data File Writer" daemon prio=10 tid=0x00007fc5b8a4e800 nid=0x55c
in Object.wait() [0x00007fc5aa8bb000]
java.lang.Thread.State: WAITING (on object monitor)
at java.lang.Object.wait(Native Method)
at java.lang.Object.wait(Object.java:503)
at
org.apache.activemq.store.kahadb.disk.journal.DataFileAppender.processQueue(DataFileAppender.java:304)
- locked <0x00000000c0bf47e8> (a java.lang.Object)
at
org.apache.activemq.store.kahadb.disk.journal.DataFileAppender$1.run(DataFileAppender.java:193)
"ActiveMQ Journal Checkpoint Worker" daemon prio=10 tid=0x00007fc5b873f800
nid=0x55b waiting on condition [0x00007fc5aa9bc000]
java.lang.Thread.State: TIMED_WAITING (sleeping)
at java.lang.Thread.sleep(Native Method)
at
org.apache.activemq.store.kahadb.AbstractKahaDBStore$2.run(AbstractKahaDBStore.java:691)
"KahaDB Scheduler" daemon prio=10 tid=0x00007fc5b8724800 nid=0x55a in
Object.wait() [0x00007fc5aaabd000]
java.lang.Thread.State: TIMED_WAITING (on object monitor)
at java.lang.Object.wait(Native Method)
at java.util.TimerThread.mainLoop(Timer.java:552)
- locked <0x00000000c11b1458> (a java.util.TaskQueue)
at java.util.TimerThread.run(Timer.java:505)
"ActiveMQ Broker[localhost] Scheduler" daemon prio=10 tid=0x00007fc5b8166000
nid=0x559 in Object.wait() [0x00007fc5aabbe000]
java.lang.Thread.State: TIMED_WAITING (on object monitor)
at java.lang.Object.wait(Native Method)
at java.util.TimerThread.mainLoop(Timer.java:552)
- locked <0x00000000c058f5e8> (a java.util.TaskQueue)
at java.util.TimerThread.run(Timer.java:505)
"RMI TCP Accept-0" daemon prio=10 tid=0x00007fc5b825a800 nid=0x551 runnable
[0x00007fc5b0151000]
java.lang.Thread.State: RUNNABLE
at java.net.PlainSocketImpl.socketAccept(Native Method)
at
java.net.AbstractPlainSocketImpl.accept(AbstractPlainSocketImpl.java:398)
at java.net.ServerSocket.implAccept(ServerSocket.java:530)
at java.net.ServerSocket.accept(ServerSocket.java:498)
at
sun.management.jmxremote.LocalRMIServerSocketFactory$1.accept(LocalRMIServerSocketFactory.java:52)
at
sun.rmi.transport.tcp.TCPTransport$AcceptLoop.executeAcceptLoop(TCPTransport.java:399)
at
sun.rmi.transport.tcp.TCPTransport$AcceptLoop.run(TCPTransport.java:371)
at java.lang.Thread.run(Thread.java:745)
"Service Thread" daemon prio=10 tid=0x00007fc5b8176800 nid=0x54d runnable
[0x0000000000000000]
java.lang.Thread.State: RUNNABLE
"C2 CompilerThread1" daemon prio=10 tid=0x00007fc5b8174000 nid=0x54c waiting
on condition [0x0000000000000000]
java.lang.Thread.State: RUNNABLE
"C2 CompilerThread0" daemon prio=10 tid=0x00007fc5b8171000 nid=0x54b waiting
on condition [0x0000000000000000]
java.lang.Thread.State: RUNNABLE
"Signal Dispatcher" daemon prio=10 tid=0x00007fc5b816f000 nid=0x549 runnable
[0x0000000000000000]
java.lang.Thread.State: RUNNABLE
"Finalizer" daemon prio=10 tid=0x00007fc5b8143800 nid=0x540 in Object.wait()
[0x00007fc5b113a000]
java.lang.Thread.State: WAITING (on object monitor)
at java.lang.Object.wait(Native Method)
at java.lang.ref.ReferenceQueue.remove(ReferenceQueue.java:135)
- locked <0x00000000c02cdfb8> (a java.lang.ref.ReferenceQueue$Lock)
at java.lang.ref.ReferenceQueue.remove(ReferenceQueue.java:151)
at java.lang.ref.Finalizer$FinalizerThread.run(Finalizer.java:209)
"Reference Handler" daemon prio=10 tid=0x00007fc5b8141800 nid=0x53f in
Object.wait() [0x00007fc5b123b000]
java.lang.Thread.State: WAITING (on object monitor)
at java.lang.Object.wait(Native Method)
at java.lang.Object.wait(Object.java:503)
at java.lang.ref.Reference$ReferenceHandler.run(Reference.java:133)
- locked <0x00000000c02cdd38> (a java.lang.ref.Reference$Lock)
"main" prio=10 tid=0x00007fc5b800a800 nid=0x534 waiting on condition
[0x00007fc5bf999000]
java.lang.Thread.State: WAITING (parking)
at sun.misc.Unsafe.park(Native Method)
- parking to wait for <0x00000000c11b2328> (a
java.util.concurrent.CountDownLatch$Sync)
at java.util.concurrent.locks.LockSupport.park(LockSupport.java:186)
at
java.util.concurrent.locks.AbstractQueuedSynchronizer.parkAndCheckInterrupt(AbstractQueuedSynchronizer.java:834)
at
java.util.concurrent.locks.AbstractQueuedSynchronizer.doAcquireSharedInterruptibly(AbstractQueuedSynchronizer.java:994)
at
java.util.concurrent.locks.AbstractQueuedSynchronizer.acquireSharedInterruptibly(AbstractQueuedSynchronizer.java:1303)
at java.util.concurrent.CountDownLatch.await(CountDownLatch.java:236)
at
org.apache.activemq.console.command.StartCommand.runTask(StartCommand.java:118)
at
org.apache.activemq.console.command.AbstractCommand.execute(AbstractCommand.java:57)
at
org.apache.activemq.console.command.ShellCommand.runTask(ShellCommand.java:150)
at
org.apache.activemq.console.command.AbstractCommand.execute(AbstractCommand.java:57)
at
org.apache.activemq.console.command.ShellCommand.main(ShellCommand.java:104)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at
sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)
at
sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:606)
at org.apache.activemq.console.Main.runTaskClass(Main.java:262)
at org.apache.activemq.console.Main.main(Main.java:115)
"VM Thread" prio=10 tid=0x00007fc5b813d000 nid=0x53e runnable
"GC task thread#0 (ParallelGC)" prio=10 tid=0x00007fc5b8020800 nid=0x535
runnable
"GC task thread#1 (ParallelGC)" prio=10 tid=0x00007fc5b8022000 nid=0x536
runnable
"GC task thread#2 (ParallelGC)" prio=10 tid=0x00007fc5b8024000 nid=0x537
runnable
"GC task thread#3 (ParallelGC)" prio=10 tid=0x00007fc5b8026000 nid=0x538
runnable
"VM Periodic Task Thread" prio=10 tid=0x00007fc5b8264000 nid=0x552 waiting
on condition
JNI global references: 146
Tim Bain wrote
> Next time it happens, see if you can browse the messages on the queues.
> That might indicate whether the problem is the data store (if you can't
> browse the queues) or something more localized like the dispatch thread
> (if
> you can).
>
> Also, you need to take thread dumps and/or use a profiler (both is best)
> to
> let us figure out what the different threads are doing. I'm out of ideas
> based on what information you've got at the moment, so hopefully seeing
> what the threads are doing will help narrow in on the problem.
>
> Tim
> On Aug 22, 2015 6:50 PM, "Richard Sinek" <
> rsinek@
> > wrote:
>
>> There is a specific queue that gets the scheduled messages and generally
>> a
>> single consumer thread on that queue. That consumer is part of the same
>> application that is pulling from all the other queues.
>>
>> We don't use selectors on any queue, of which there are 15, nor do we
>> have
>> any destination based GC.
>>
>> The web console is fully responsive during the events, I have not
>> actually
>> tried browsing.
>>
>> The issue is very repeatable, in many cases happening multiple times per
>> day. Often it gets progressively worse and restarting the consumers
>> usually
>> helps.
>>
>>
>>
>>
>>
>>
>>
>> --
>> View this message in context:
>> http://activemq.2283324.n4.nabble.com/AMQ-pauses-sending-to-consumers-tp4701242p4701309.html
>> Sent from the ActiveMQ - User mailing list archive at Nabble.com.
>>
--
View this message in context:
http://activemq.2283324.n4.nabble.com/AMQ-pauses-sending-to-consumers-tp4701242p4701634.html
Sent from the ActiveMQ - User mailing list archive at Nabble.com.
