One of our camel applications just stopped working correctly and I found the
below exception in then log.
The application did not die, it carried on running and consuming messages,
but it stopped writing to the mina component that sends messages to our
splunk server over mina:tcp.
The error looks like it is coming from activemq because it mentions Mysql
and our application doesnt use mysql.
Could a single route fail/thread be crashing silently? Is there a way to
make sure that the application dies if anything like this happens so we know
to restart it..
I'm not sure if it is helpful but I have put a thread dump of when it was
not running correctly at the bottom of this email..
@400000004dc7b36903d4eb3c 2011-05-09 10:26:55,062 [Camel (mrs) thread #24 -
JmsConsumer[general.in]] INFO org.apache.activemq.TransactionContext -
commit failed for transaction
TX:ID:vp-prod-wfe-i-39469-1304763938202-0:5:173635
@400000004dc7b36903d4f6f4 javax.jms.JMSException: STORE COMMIT FAILED:
Transaction rolled back.
@400000004dc7b36903d4fadc at
org.apache.activemq.util.JMSExceptionSupport.create(JMSExceptionSupport.java:49)
@400000004dc7b36903d502ac at
org.apache.activemq.ActiveMQConnection.syncSendPacket(ActiveMQConnection.java:1287)
@400000004dc7b36903d5b65c at
org.apache.activemq.TransactionContext.syncSendPacketWithInterruptionHandling(TransactionContext.java:704)
@400000004dc7b36903d5ba44 at
org.apache.activemq.TransactionContext.commit(TransactionContext.java:299)
@400000004dc7b36903d5be2c at
org.apache.activemq.ActiveMQSession.commit(ActiveMQSession.java:560)
@400000004dc7b36903d5c214 at
org.apache.activemq.pool.PooledSession.commit(PooledSession.java:133)
@400000004dc7b36903d744cc at
org.springframework.jms.support.JmsUtils.commitIfNecessary(JmsUtils.java:217)
@400000004dc7b36903d74c9c at
org.springframework.jms.listener.AbstractMessageListenerContainer.commitIfNecessary(AbstractMessageListenerContainer.java:575)
@400000004dc7b36903d75084 at
org.springframework.jms.listener.AbstractMessageListenerContainer.doExecuteListener(AbstractMessageListenerContainer.java:481)
@400000004dc7b36903d77b7c at
org.springframework.jms.listener.AbstractPollingMessageListenerContainer.doReceiveAndExecute(AbstractPollingMessageListenerContainer.java:325)
@400000004dc7b36903d77f64 at
org.springframework.jms.listener.AbstractPollingMessageListenerContainer.receiveAndExecute(AbstractPollingMessageListenerContainer.java:243)
@400000004dc7b36903d7834c at
org.springframework.jms.listener.DefaultMessageListenerContainer$AsyncMessageListenerInvoker.invokeListener(DefaultMessageListenerContainer.java:1058)
@400000004dc7b36903d796d4 at
org.springframework.jms.listener.DefaultMessageListenerContainer$AsyncMessageListenerInvoker.executeOngoingLoop(DefaultMessageListenerContainer.java:1050)
@400000004dc7b36903d79abc at
org.springframework.jms.listener.DefaultMessageListenerContainer$AsyncMessageListenerInvoker.run(DefaultMessageListenerContainer.java:947)
@400000004dc7b36903d7ae44 at
java.util.concurrent.ThreadPoolExecutor$Worker.runTask(Unknown Source)
@400000004dc7b36903d7b22c at
java.util.concurrent.ThreadPoolExecutor$Worker.run(Unknown Source)
@400000004dc7b36903d7b614 at java.lang.Thread.run(Unknown Source)
@400000004dc7b36903d7d16c Caused by: javax.transaction.xa.XAException: STORE
COMMIT FAILED: Transaction rolled back.
@400000004dc7b36903d7d554 at
org.apache.activemq.transaction.LocalTransaction.commit(LocalTransaction.java:77)
@400000004dc7b36903d7d93c at
org.apache.activemq.broker.TransactionBroker.commitTransaction(TransactionBroker.java:173)
@400000004dc7b36903d7dd24 at
org.apache.activemq.broker.MutableBrokerFilter.commitTransaction(MutableBrokerFilter.java:103)
@400000004dc7b36903d80fec at
org.apache.activemq.broker.TransportConnection.processCommitTransactionOnePhase(TransportConnection.java:420)
@400000004dc7b36903d813d4 at
org.apache.activemq.command.TransactionInfo.visit(TransactionInfo.java:100)
@400000004dc7b36903d817bc at
org.apache.activemq.broker.TransportConnection.service(TransportConnection.java:306)
@400000004dc7b36903d81ba4 at
org.apache.activemq.broker.TransportConnection$1.onCommand(TransportConnection.java:179)
@400000004dc7b36903d82b44 at
org.apache.activemq.transport.TransportFilter.onCommand(TransportFilter.java:69)
@400000004dc7b36903d82f2c at
org.apache.activemq.transport.WireFormatNegotiator.onCommand(WireFormatNegotiator.java:113)
@400000004dc7b36903d83314 at
org.apache.activemq.transport.InactivityMonitor.onCommand(InactivityMonitor.java:227)
@400000004dc7b36903d85254 at
org.apache.activemq.transport.TransportSupport.doConsume(TransportSupport.java:83)
@400000004dc7b36903d8563c at
org.apache.activemq.transport.tcp.TcpTransport.doRun(TcpTransport.java:220)
@400000004dc7b36903d85a24 at
org.apache.activemq.transport.tcp.TcpTransport.run(TcpTransport.java:202)
@400000004dc7b36903d85e0c ... 1 more
@400000004dc7b36903d85e0c Caused by: java.io.IOException: Failed to get
store sequenceId for messageId:
ID:vp-prod-wfe-j-40014-1304519813061-0:13:5:1:1385, on: queue://general.in.
Reason:
com.mysql.jdbc.exceptions.jdbc4.MySQLNonTransientConnectionException: No
operations allowed after statement closed.
@400000004dc7b36903d87194 at
org.apache.activemq.util.IOExceptionSupport.create(IOExceptionSupport.java:33)
@400000004dc7b36903d8757c at
org.apache.activemq.store.jdbc.JDBCMessageStore.getStoreSequenceIdForMessageId(JDBCMessageStore.java:318)
@400000004dc7b36903d8a45c at
org.apache.activemq.store.jdbc.JDBCMessageStore.removeMessage(JDBCMessageStore.java:170)
@400000004dc7b36903d8a844 at
org.apache.activemq.store.memory.MemoryTransactionStore$4.run(MemoryTransactionStore.java:306)
@400000004dc7b36903d8ac2c at
org.apache.activemq.store.memory.MemoryTransactionStore$Tx.commit(MemoryTransactionStore.java:103)
@400000004dc7b36903d8cb6c at
org.apache.activemq.store.memory.MemoryTransactionStore.commit(MemoryTransactionStore.java:226)
@400000004dc7b36903d8cf54 at
org.apache.activemq.transaction.LocalTransaction.commit(LocalTransaction.java:72)
@400000004dc7b36903d8d33c ... 13 more
@400000004dc7b36903d8d724 Caused by: java.lang.Throwable:
com.mysql.jdbc.exceptions.jdbc4.MySQLNonTransientConnectionException: No
operations allowed after statement closed.
@400000004dc7b36903d8eaac at
sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native Method)
@400000004dc7b36903d8ee94 at
sun.reflect.NativeConstructorAccessorImpl.newInstance(Unknown Source)
@400000004dc7b36903d8ee94 at
sun.reflect.DelegatingConstructorAccessorImpl.newInstance(Unknown Source)
@400000004dc7b36903d8f27c at
java.lang.reflect.Constructor.newInstance(Unknown Source)
@400000004dc7b36903d9021c at
com.mysql.jdbc.Util.handleNewInstance(Util.java:407)
@400000004dc7b36903d90604 at
com.mysql.jdbc.Util.getInstance(Util.java:382)
@400000004dc7b36903d9292c at
com.mysql.jdbc.SQLError.createSQLException(SQLError.java:1013)
@400000004dc7b36903d92d14 at
com.mysql.jdbc.SQLError.createSQLException(SQLError.java:987)
@400000004dc7b36903d930fc at
com.mysql.jdbc.SQLError.createSQLException(SQLError.java:982)
@400000004dc7b36903d94484 at
com.mysql.jdbc.SQLError.createSQLException(SQLError.java:927)
@400000004dc7b36903d9486c at
com.mysql.jdbc.StatementImpl.checkClosed(StatementImpl.java:402)
@400000004dc7b36903d967ac at
com.mysql.jdbc.PreparedStatement.setString(PreparedStatement.java:4394)
@400000004dc7b36903d96b94 at
org.apache.commons.dbcp.DelegatingPreparedStatement.setString(DelegatingPreparedStatement.java:132)
@400000004dc7b36903d96f7c at
org.apache.commons.dbcp.DelegatingPreparedStatement.setString(DelegatingPreparedStatement.java:132)
@400000004dc7b36903d97364 at
org.apache.activemq.store.jdbc.adapter.DefaultJDBCAdapter.getStoreSequenceId(DefaultJDBCAdapter.java:272)
@400000004dc7b36903dc2e9c at
org.apache.activemq.store.jdbc.JDBCMessageStore.getStoreSequenceIdForMessageId(JDBCMessageStore.java:315)
@400000004dc7b36903dc366c ... 18 more
@400000004dc7b36903de594c 2011-05-09 10:26:55,064 [Camel (mrs) thread #24 -
JmsConsumer[general.in]] WARN
org.apache.camel.component.jms.JmsMessageListenerContainer - Execution of
JMS message listener failed, and no ErrorHandler has been set.
@400000004dc7b36903de6504 javax.jms.JMSException: STORE COMMIT FAILED:
Transaction rolled back.
@400000004dc7b36903de68ec at
org.apache.activemq.util.JMSExceptionSupport.create(JMSExceptionSupport.java:49)
@400000004dc7b36903de6cd4 at
org.apache.activemq.ActiveMQConnection.syncSendPacket(ActiveMQConnection.java:1287)
@400000004dc7b36903de8c14 at
org.apache.activemq.TransactionContext.syncSendPacketWithInterruptionHandling(TransactionContext.java:704)
@400000004dc7b36903de8ffc at
org.apache.activemq.TransactionContext.commit(TransactionContext.java:299)
@400000004dc7b36903de93e4 at
org.apache.activemq.ActiveMQSession.commit(ActiveMQSession.java:560)
@400000004dc7b36903de97cc at
org.apache.activemq.pool.PooledSession.commit(PooledSession.java:133)
@400000004dc7b36903dea76c at
org.springframework.jms.support.JmsUtils.commitIfNecessary(JmsUtils.java:217)
@400000004dc7b36903deab54 at
org.springframework.jms.listener.AbstractMessageListenerContainer.commitIfNecessary(AbstractMessageListenerContainer.java:575)
@400000004dc7b36903deaf3c at
org.springframework.jms.listener.AbstractMessageListenerContainer.doExecuteListener(AbstractMessageListenerContainer.java:481)
@400000004dc7b36903dece7c at
org.springframework.jms.listener.AbstractPollingMessageListenerContainer.doReceiveAndExecute(AbstractPollingMessageListenerContainer.java:325)
@400000004dc7b36903ded264 at
org.springframework.jms.listener.AbstractPollingMessageListenerContainer.receiveAndExecute(AbstractPollingMessageListenerContainer.java:243)
@400000004dc7b36903deda34 at
org.springframework.jms.listener.DefaultMessageListenerContainer$AsyncMessageListenerInvoker.invokeListener(DefaultMessageListenerContainer.java:1058)
@400000004dc7b36903dfd434 at
org.springframework.jms.listener.DefaultMessageListenerContainer$AsyncMessageListenerInvoker.executeOngoingLoop(DefaultMessageListenerContainer.java:1050)
@400000004dc7b36903dfdc04 at
org.springframework.jms.listener.DefaultMessageListenerContainer$AsyncMessageListenerInvoker.run(DefaultMessageListenerContainer.java:947)
@400000004dc7b36903dff374 at
java.util.concurrent.ThreadPoolExecutor$Worker.runTask(Unknown Source)
@400000004dc7b36903dff75c at
java.util.concurrent.ThreadPoolExecutor$Worker.run(Unknown Source)
@400000004dc7b36903dffb44 at java.lang.Thread.run(Unknown Source)
@400000004dc7b36903e01a84 Caused by: javax.transaction.xa.XAException: STORE
COMMIT FAILED: Transaction rolled back.
@400000004dc7b36903e01e6c at
org.apache.activemq.transaction.LocalTransaction.commit(LocalTransaction.java:77)
@400000004dc7b36903e02254 at
org.apache.activemq.broker.TransactionBroker.commitTransaction(TransactionBroker.java:173)
@400000004dc7b36903e0263c at
org.apache.activemq.broker.MutableBrokerFilter.commitTransaction(MutableBrokerFilter.java:103)
@400000004dc7b36903e035dc at
org.apache.activemq.broker.TransportConnection.processCommitTransactionOnePhase(TransportConnection.java:420)
@400000004dc7b36903e03dac at
org.apache.activemq.command.TransactionInfo.visit(TransactionInfo.java:100)
@400000004dc7b36903e04194 at
org.apache.activemq.broker.TransportConnection.service(TransportConnection.java:306)
@400000004dc7b36903e0457c at
org.apache.activemq.broker.TransportConnection$1.onCommand(TransportConnection.java:179)
@400000004dc7b36903e07844 at
org.apache.activemq.transport.TransportFilter.onCommand(TransportFilter.java:69)
@400000004dc7b36903e07c2c at
org.apache.activemq.transport.WireFormatNegotiator.onCommand(WireFormatNegotiator.java:113)
@400000004dc7b36903e08014 at
org.apache.activemq.transport.InactivityMonitor.onCommand(InactivityMonitor.java:227)
@400000004dc7b36903e0c664 at
org.apache.activemq.transport.TransportSupport.doConsume(TransportSupport.java:83)
@400000004dc7b36903e0ca4c at
org.apache.activemq.transport.tcp.TcpTransport.doRun(TcpTransport.java:220)
@400000004dc7b36903e0ce34 at
org.apache.activemq.transport.tcp.TcpTransport.run(TcpTransport.java:202)
@400000004dc7b36903e0d21c ... 1 more
@400000004dc7b36903e0d21c Caused by: java.io.IOException: Failed to get
store sequenceId for messageId:
ID:vp-prod-wfe-j-40014-1304519813061-0:13:5:1:1385, on: queue://general.in.
Reason:
com.mysql.jdbc.exceptions.jdbc4.MySQLNonTransientConnectionException: No
operations allowed after statement closed.
@400000004dc7b36903e10cb4 at
org.apache.activemq.util.IOExceptionSupport.create(IOExceptionSupport.java:33)
@400000004dc7b36903e1109c at
org.apache.activemq.store.jdbc.JDBCMessageStore.getStoreSequenceIdForMessageId(JDBCMessageStore.java:318)
@400000004dc7b36903e1203c at
org.apache.activemq.store.jdbc.JDBCMessageStore.removeMessage(JDBCMessageStore.java:170)
@400000004dc7b36903e12424 at
org.apache.activemq.store.memory.MemoryTransactionStore$4.run(MemoryTransactionStore.java:306)
@400000004dc7b36903e1280c at
org.apache.activemq.store.memory.MemoryTransactionStore$Tx.commit(MemoryTransactionStore.java:103)
@400000004dc7b36903e1474c at
org.apache.activemq.store.memory.MemoryTransactionStore.commit(MemoryTransactionStore.java:226)
@400000004dc7b36903e14b34 at
org.apache.activemq.transaction.LocalTransaction.commit(LocalTransaction.java:72)
@400000004dc7b36903e14f1c ... 13 more
@400000004dc7b36903e15304 Caused by: java.lang.Throwable:
com.mysql.jdbc.exceptions.jdbc4.MySQLNonTransientConnectionException: No
operations allowed after statement closed.
@400000004dc7b36903e162a4 at
sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native Method)
@400000004dc7b36903e1668c at
sun.reflect.NativeConstructorAccessorImpl.newInstance(Unknown Source)
@400000004dc7b36903e16a74 at
sun.reflect.DelegatingConstructorAccessorImpl.newInstance(Unknown Source)
@400000004dc7b36903e16e5c at
java.lang.reflect.Constructor.newInstance(Unknown Source)
@400000004dc7b36903e189b4 at
com.mysql.jdbc.Util.handleNewInstance(Util.java:407)
@400000004dc7b36903e18d9c at
com.mysql.jdbc.Util.getInstance(Util.java:382)
@400000004dc7b36903e18d9c at
com.mysql.jdbc.SQLError.createSQLException(SQLError.java:1013)
@400000004dc7b36903e19184 at
com.mysql.jdbc.SQLError.createSQLException(SQLError.java:987)
@400000004dc7b36903e1956c at
com.mysql.jdbc.SQLError.createSQLException(SQLError.java:982)
@400000004dc7b36903e1a50c at
com.mysql.jdbc.SQLError.createSQLException(SQLError.java:927)
@400000004dc7b36903e1c44c at
com.mysql.jdbc.StatementImpl.checkClosed(StatementImpl.java:402)
@400000004dc7b36903e1c834 at
com.mysql.jdbc.PreparedStatement.setString(PreparedStatement.java:4394)
@400000004dc7b36903e1cc1c at
org.apache.commons.dbcp.DelegatingPreparedStatement.setString(DelegatingPreparedStatement.java:132)
@400000004dc7b36903e1d004 at
org.apache.commons.dbcp.DelegatingPreparedStatement.setString(DelegatingPreparedStatement.java:132)
@400000004dc7b36903e1dfa4 at
org.apache.activemq.store.jdbc.adapter.DefaultJDBCAdapter.getStoreSequenceId(DefaultJDBCAdapter.java:272)
@400000004dc7b36903e1e38c at
org.apache.activemq.store.jdbc.JDBCMessageStore.getStoreSequenceIdForMessageId(JDBCMessageStore.java:315)
@400000004dc7b36903e1eb5c ... 18 more
@400000004dc7b36903f18774 2011-05-09 10:26:55,066 [Camel (mrs) thread #24 -
JmsConsumer[general.in]] WARN
org.apache.camel.component.jms.JmsMessageListenerContainer - Setup of JMS
message listener invoker failed for destination 'general.in' - trying to
recover. Cause: STORE COMMIT FAILED: Transaction rolled back.
Full thread dump Java HotSpot(TM) Server VM (19.0-b09 mixed mode):
"Camel (mrs) thread #29 - JmsConsumer[general.in]" daemon prio=3
tid=0x09975000 nid=0x90 in Object.wait() [0xb5e19000]
java.lang.Thread.State: TIMED_WAITING (on object monitor)
at java.lang.Object.wait(Native Method)
at
org.apache.activemq.SimplePriorityMessageDispatchChannel.dequeue(SimplePriorityMessageDispatchChannel.java:89)
- locked <0xbb49b270> (a java.lang.Object)
at
org.apache.activemq.ActiveMQMessageConsumer.dequeue(ActiveMQMessageConsumer.java:452)
at
org.apache.activemq.ActiveMQMessageConsumer.receive(ActiveMQMessageConsumer.java:577)
at
org.springframework.jms.listener.AbstractPollingMessageListenerContainer.receiveMessage(AbstractPollingMessageListenerContainer.java:429)
at
org.springframework.jms.listener.AbstractPollingMessageListenerContainer.doReceiveAndExecute(AbstractPollingMessageListenerContainer.java:310)
at
org.springframework.jms.listener.AbstractPollingMessageListenerContainer.receiveAndExecute(AbstractPollingMessageListenerContainer.java:243)
at
org.springframework.jms.listener.DefaultMessageListenerContainer$AsyncMessageListenerInvoker.invokeListener(DefaultMessageListenerContainer.java:1058)
at
org.springframework.jms.listener.DefaultMessageListenerContainer$AsyncMessageListenerInvoker.executeOngoingLoop(DefaultMessageListenerContainer.java:1050)
at
org.springframework.jms.listener.DefaultMessageListenerContainer$AsyncMessageListenerInvoker.run(DefaultMessageListenerContainer.java:947)
at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(Unknown
Source)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(Unknown Source)
at java.lang.Thread.run(Unknown Source)
"Camel (mrs) thread #28 - ErrorHandlerRedeliveryTask" daemon prio=3
tid=0x098b9000 nid=0x8f waiting on condition [0xb51d5000]
java.lang.Thread.State: TIMED_WAITING (parking)
at sun.misc.Unsafe.park(Native Method)
- parking to wait for <0xbb1c3338> (a
java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
at java.util.concurrent.locks.LockSupport.parkNanos(Unknown Source)
at
java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(Unknown
Source)
at java.util.concurrent.DelayQueue.take(Unknown Source)
at
java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(Unknown
Source)
at
java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(Unknown
Source)
at java.util.concurrent.ThreadPoolExecutor.getTask(Unknown Source)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(Unknown Source)
at java.lang.Thread.run(Unknown Source)
"Camel (mrs) thread #27 - MinaSocketConnector" prio=3 tid=0x08ae6800
nid=0x8e waiting on condition [0xb5184000]
java.lang.Thread.State: WAITING (parking)
at sun.misc.Unsafe.park(Native Method)
- parking to wait for <0xbb1ac2d0> (a
java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
at java.util.concurrent.locks.LockSupport.park(Unknown Source)
at
java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(Unknown
Source)
at java.util.concurrent.LinkedBlockingQueue.take(Unknown Source)
at java.util.concurrent.ThreadPoolExecutor.getTask(Unknown Source)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(Unknown Source)
at java.lang.Thread.run(Unknown Source)
"ActiveMQ Session Task" prio=3 tid=0x0835b800 nid=0x8d waiting on condition
[0xb5f0c000]
java.lang.Thread.State: TIMED_WAITING (parking)
at sun.misc.Unsafe.park(Native Method)
- parking to wait for <0xbb484c20> (a
java.util.concurrent.SynchronousQueue$TransferStack)
at java.util.concurrent.locks.LockSupport.parkNanos(Unknown Source)
at
java.util.concurrent.SynchronousQueue$TransferStack.awaitFulfill(Unknown
Source)
at java.util.concurrent.SynchronousQueue$TransferStack.transfer(Unknown
Source)
at java.util.concurrent.SynchronousQueue.poll(Unknown Source)
at java.util.concurrent.ThreadPoolExecutor.getTask(Unknown Source)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(Unknown Source)
at java.lang.Thread.run(Unknown Source)
"ActiveMQ Session Task" prio=3 tid=0x08957400 nid=0x8c waiting on condition
[0xb5133000]
java.lang.Thread.State: TIMED_WAITING (parking)
at sun.misc.Unsafe.park(Native Method)
- parking to wait for <0xbb484c20> (a
java.util.concurrent.SynchronousQueue$TransferStack)
at java.util.concurrent.locks.LockSupport.parkNanos(Unknown Source)
at
java.util.concurrent.SynchronousQueue$TransferStack.awaitFulfill(Unknown
Source)
at java.util.concurrent.SynchronousQueue$TransferStack.transfer(Unknown
Source)
at java.util.concurrent.SynchronousQueue.poll(Unknown Source)
at java.util.concurrent.ThreadPoolExecutor.getTask(Unknown Source)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(Unknown Source)
at java.lang.Thread.run(Unknown Source)
"InactivityMonitor WriteCheck" daemon prio=3 tid=0x09974c00 nid=0x89 in
Object.wait() [0xb52c8000]
java.lang.Thread.State: TIMED_WAITING (on object monitor)
at java.lang.Object.wait(Native Method)
at java.util.TimerThread.mainLoop(Unknown Source)
- locked <0xbb485b30> (a java.util.TaskQueue)
at java.util.TimerThread.run(Unknown Source)
"InactivityMonitor ReadCheck" daemon prio=3 tid=0x08b0fc00 nid=0x88 in
Object.wait() [0xb545d000]
java.lang.Thread.State: TIMED_WAITING (on object monitor)
at java.lang.Object.wait(Native Method)
at java.util.TimerThread.mainLoop(Unknown Source)
- locked <0xbb485f10> (a java.util.TaskQueue)
at java.util.TimerThread.run(Unknown Source)
"Camel (mrs) thread #26 - JmsConsumer[splunk.in]" daemon prio=3
tid=0x081c6000 nid=0x87 waiting on condition [0xb5ebb000]
java.lang.Thread.State: WAITING (parking)
at sun.misc.Unsafe.park(Native Method)
- parking to wait for <0xbb4a3378> (a
java.util.concurrent.CountDownLatch$Sync)
at java.util.concurrent.locks.LockSupport.park(Unknown Source)
at
java.util.concurrent.locks.AbstractQueuedSynchronizer.parkAndCheckInterrupt(Unknown
Source)
at
java.util.concurrent.locks.AbstractQueuedSynchronizer.doAcquireSharedInterruptibly(Unknown
Source)
at
java.util.concurrent.locks.AbstractQueuedSynchronizer.acquireSharedInterruptibly(Unknown
Source)
at java.util.concurrent.CountDownLatch.await(Unknown Source)
at
org.apache.camel.util.AsyncProcessorHelper.process(AsyncProcessorHelper.java:122)
at
org.apache.camel.processor.DelegateAsyncProcessor.process(DelegateAsyncProcessor.java:85)
at
org.apache.camel.component.jms.EndpointMessageListener.onMessage(EndpointMessageListener.java:92)
at
org.springframework.jms.listener.AbstractMessageListenerContainer.doInvokeListener(AbstractMessageListenerContainer.java:560)
at
org.springframework.jms.listener.AbstractMessageListenerContainer.invokeListener(AbstractMessageListenerContainer.java:498)
at
org.springframework.jms.listener.AbstractMessageListenerContainer.doExecuteListener(AbstractMessageListenerContainer.java:467)
at
org.springframework.jms.listener.AbstractPollingMessageListenerContainer.doReceiveAndExecute(AbstractPollingMessageListenerContainer.java:325)
at
org.springframework.jms.listener.AbstractPollingMessageListenerContainer.receiveAndExecute(AbstractPollingMessageListenerContainer.java:243)
at
org.springframework.jms.listener.DefaultMessageListenerContainer$AsyncMessageListenerInvoker.invokeListener(DefaultMessageListenerContainer.java:1058)
at
org.springframework.jms.listener.DefaultMessageListenerContainer$AsyncMessageListenerInvoker.executeOngoingLoop(DefaultMessageListenerContainer.java:1050)
at
org.springframework.jms.listener.DefaultMessageListenerContainer$AsyncMessageListenerInvoker.run(DefaultMessageListenerContainer.java:947)
at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(Unknown
Source)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(Unknown Source)
at java.lang.Thread.run(Unknown Source)
"ActiveMQ Transport: tcp://
nm-prod-wfe-mq.bbc.redbeemedia.net/10.232.1.248:61616" prio=3 tid=0x081de400
nid=0x86 runnable [0xb5f5d000]
java.lang.Thread.State: RUNNABLE
at java.net.SocketInputStream.socketRead0(Native Method)
at java.net.SocketInputStream.read(Unknown Source)
at
org.apache.activemq.transport.tcp.TcpBufferedInputStream.fill(TcpBufferedInputStream.java:50)
at
org.apache.activemq.transport.tcp.TcpTransport$2.fill(TcpTransport.java:576)
at
org.apache.activemq.transport.tcp.TcpBufferedInputStream.read(TcpBufferedInputStream.java:58)
at
org.apache.activemq.transport.tcp.TcpTransport$2.read(TcpTransport.java:561)
at java.io.DataInputStream.readInt(Unknown Source)
at
org.apache.activemq.openwire.OpenWireFormat.unmarshal(OpenWireFormat.java:269)
at
org.apache.activemq.transport.tcp.TcpTransport.readCommand(TcpTransport.java:227)
at
org.apache.activemq.transport.tcp.TcpTransport.doRun(TcpTransport.java:219)
at
org.apache.activemq.transport.tcp.TcpTransport.run(TcpTransport.java:202)
at java.lang.Thread.run(Unknown Source)
"ActiveMQConnection[ID:vp-prod-wfe-i-39469-1304763938202-0:5] Scheduler"
daemon prio=3 tid=0x08bf3c00 nid=0x85 in Object.wait() [0xb536a000]
java.lang.Thread.State: WAITING (on object monitor)
at java.lang.Object.wait(Native Method)
at java.lang.Object.wait(Object.java:485)
at java.util.TimerThread.mainLoop(Unknown Source)
- locked <0xbb488310> (a java.util.TaskQueue)
at java.util.TimerThread.run(Unknown Source)
"Camel (mrs) thread #24 - JmsConsumer[general.in]" daemon prio=3
tid=0x08458000 nid=0x7c waiting on condition [0xb5226000]
java.lang.Thread.State: TIMED_WAITING (parking)
at sun.misc.Unsafe.park(Native Method)
- parking to wait for <0xbb1c3100> (a
java.util.concurrent.SynchronousQueue$TransferStack)
at java.util.concurrent.locks.LockSupport.parkNanos(Unknown Source)
at
java.util.concurrent.SynchronousQueue$TransferStack.awaitFulfill(Unknown
Source)
at java.util.concurrent.SynchronousQueue$TransferStack.transfer(Unknown
Source)
at java.util.concurrent.SynchronousQueue.poll(Unknown Source)
at java.util.concurrent.ThreadPoolExecutor.getTask(Unknown Source)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(Unknown Source)
at java.lang.Thread.run(Unknown Source)
"Camel (mrs) thread #23 - JmsConsumer[ibroadcast.in]" daemon prio=3
tid=0x08592400 nid=0x7b in Object.wait() [0xb5277000]
java.lang.Thread.State: TIMED_WAITING (on object monitor)
at java.lang.Object.wait(Native Method)
at
org.apache.activemq.SimplePriorityMessageDispatchChannel.dequeue(SimplePriorityMessageDispatchChannel.java:89)
- locked <0xbb4910d8> (a java.lang.Object)
at
org.apache.activemq.ActiveMQMessageConsumer.dequeue(ActiveMQMessageConsumer.java:452)
at
org.apache.activemq.ActiveMQMessageConsumer.receive(ActiveMQMessageConsumer.java:577)
at
org.springframework.jms.listener.AbstractPollingMessageListenerContainer.receiveMessage(AbstractPollingMessageListenerContainer.java:429)
at
org.springframework.jms.listener.AbstractPollingMessageListenerContainer.doReceiveAndExecute(AbstractPollingMessageListenerContainer.java:310)
at
org.springframework.jms.listener.AbstractPollingMessageListenerContainer.receiveAndExecute(AbstractPollingMessageListenerContainer.java:243)
at
org.springframework.jms.listener.DefaultMessageListenerContainer$AsyncMessageListenerInvoker.invokeListener(DefaultMessageListenerContainer.java:1058)
at
org.springframework.jms.listener.DefaultMessageListenerContainer$AsyncMessageListenerInvoker.executeOngoingLoop(DefaultMessageListenerContainer.java:1050)
at
org.springframework.jms.listener.DefaultMessageListenerContainer$AsyncMessageListenerInvoker.run(DefaultMessageListenerContainer.java:947)
at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(Unknown
Source)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(Unknown Source)
at java.lang.Thread.run(Unknown Source)
"Camel (mrs) thread #22 - MinaSocketConnector" prio=3 tid=0x08d59400
nid=0x46 waiting on condition [0xb5319000]
java.lang.Thread.State: WAITING (parking)
at sun.misc.Unsafe.park(Native Method)
- parking to wait for <0xbb1ac2d0> (a
java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
at java.util.concurrent.locks.LockSupport.park(Unknown Source)
at
java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(Unknown
Source)
at java.util.concurrent.LinkedBlockingQueue.take(Unknown Source)
at java.util.concurrent.ThreadPoolExecutor.getTask(Unknown Source)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(Unknown Source)
at java.lang.Thread.run(Unknown Source)
"Camel (mrs) thread #21 - ErrorHandlerRedeliveryTask" daemon prio=3
tid=0x08b02800 nid=0x44 waiting on condition [0xb53bb000]
java.lang.Thread.State: TIMED_WAITING (parking)
at sun.misc.Unsafe.park(Native Method)
- parking to wait for <0xbb1c3338> (a
java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
at java.util.concurrent.locks.LockSupport.parkNanos(Unknown Source)
at
java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(Unknown
Source)
at java.util.concurrent.DelayQueue.take(Unknown Source)
at
java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(Unknown
Source)
at
java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(Unknown
Source)
at java.util.concurrent.ThreadPoolExecutor.getTask(Unknown Source)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(Unknown Source)
at java.lang.Thread.run(Unknown Source)
"Camel (mrs) thread #20 - ErrorHandlerRedeliveryTask" daemon prio=3
tid=0x08b01c00 nid=0x43 waiting on condition [0xb540c000]
java.lang.Thread.State: TIMED_WAITING (parking)
at sun.misc.Unsafe.park(Native Method)
- parking to wait for <0xbb1c3338> (a
java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
at java.util.concurrent.locks.LockSupport.parkNanos(Unknown Source)
at
java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(Unknown
Source)
at java.util.concurrent.DelayQueue.take(Unknown Source)
at
java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(Unknown
Source)
at
java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(Unknown
Source)
at java.util.concurrent.ThreadPoolExecutor.getTask(Unknown Source)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(Unknown Source)
at java.lang.Thread.run(Unknown Source)
"Camel (mrs) thread #19 - ErrorHandlerRedeliveryTask" daemon prio=3
tid=0x08d57c00 nid=0x42 waiting on condition [0xb5936000]
java.lang.Thread.State: TIMED_WAITING (parking)
at sun.misc.Unsafe.park(Native Method)
- parking to wait for <0xbb1c3338> (a
java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
at java.util.concurrent.locks.LockSupport.parkNanos(Unknown Source)
at
java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(Unknown
Source)
at java.util.concurrent.DelayQueue.take(Unknown Source)
at
java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(Unknown
Source)
at
java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(Unknown
Source)
at java.util.concurrent.ThreadPoolExecutor.getTask(Unknown Source)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(Unknown Source)
at java.lang.Thread.run(Unknown Source)
"Camel (mrs) thread #18 - ErrorHandlerRedeliveryTask" daemon prio=3
tid=0x08c37400 nid=0x41 waiting on condition [0xb5d77000]
java.lang.Thread.State: TIMED_WAITING (parking)
at sun.misc.Unsafe.park(Native Method)
- parking to wait for <0xbb1c3338> (a
java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
at java.util.concurrent.locks.LockSupport.parkNanos(Unknown Source)
at
java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(Unknown
Source)
at java.util.concurrent.DelayQueue.take(Unknown Source)
at
java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(Unknown
Source)
at
java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(Unknown
Source)
at java.util.concurrent.ThreadPoolExecutor.getTask(Unknown Source)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(Unknown Source)
at java.lang.Thread.run(Unknown Source)
"Camel (mrs) thread #17 - MinaSocketConnector" prio=3 tid=0x09210400
nid=0x40 waiting on condition [0xb5d26000]
java.lang.Thread.State: WAITING (parking)
at sun.misc.Unsafe.park(Native Method)
- parking to wait for <0xbb1ac2d0> (a
java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
at java.util.concurrent.locks.LockSupport.park(Unknown Source)
at
java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(Unknown
Source)
at java.util.concurrent.LinkedBlockingQueue.take(Unknown Source)
at java.util.concurrent.ThreadPoolExecutor.getTask(Unknown Source)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(Unknown Source)
at java.lang.Thread.run(Unknown Source)
"Camel (mrs) thread #15 - MinaThreadPool" prio=3 tid=0x0ae90000 nid=0x38
waiting on condition [0xb5513000]
java.lang.Thread.State: WAITING (parking)
at sun.misc.Unsafe.park(Native Method)
- parking to wait for <0xbb1abfd0> (a
java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
at java.util.concurrent.locks.LockSupport.park(Unknown Source)
at
java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(Unknown
Source)
at java.util.concurrent.LinkedBlockingQueue.take(Unknown Source)
at java.util.concurrent.ThreadPoolExecutor.getTask(Unknown Source)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(Unknown Source)
at java.lang.Thread.run(Unknown Source)
"Camel (mrs) thread #14 - MinaThreadPool" prio=3 tid=0x0ae8dc00 nid=0x37
waiting on condition [0xb5564000]
java.lang.Thread.State: WAITING (parking)
at sun.misc.Unsafe.park(Native Method)
- parking to wait for <0xbb1abfd0> (a
java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
at java.util.concurrent.locks.LockSupport.park(Unknown Source)
at
java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(Unknown
Source)
at java.util.concurrent.LinkedBlockingQueue.take(Unknown Source)
at java.util.concurrent.ThreadPoolExecutor.getTask(Unknown Source)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(Unknown Source)
at java.lang.Thread.run(Unknown Source)
"Camel (mrs) thread #13 - MinaThreadPool" prio=3 tid=0x0a8cd000 nid=0x36
waiting on condition [0xb55b5000]
java.lang.Thread.State: WAITING (parking)
at sun.misc.Unsafe.park(Native Method)
- parking to wait for <0xbb1abfd0> (a
java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
at java.util.concurrent.locks.LockSupport.park(Unknown Source)
at
java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(Unknown
Source)
at java.util.concurrent.LinkedBlockingQueue.take(Unknown Source)
at java.util.concurrent.ThreadPoolExecutor.getTask(Unknown Source)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(Unknown Source)
at java.lang.Thread.run(Unknown Source)
"Camel (mrs) thread #12 - MinaThreadPool" prio=3 tid=0x0af25400 nid=0x35
waiting on condition [0xb5606000]
java.lang.Thread.State: WAITING (parking)
at sun.misc.Unsafe.park(Native Method)
- parking to wait for <0xbb1abfd0> (a
java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
at java.util.concurrent.locks.LockSupport.park(Unknown Source)
at
java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(Unknown
Source)
at java.util.concurrent.LinkedBlockingQueue.take(Unknown Source)
at java.util.concurrent.ThreadPoolExecutor.getTask(Unknown Source)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(Unknown Source)
at java.lang.Thread.run(Unknown Source)
"Camel (mrs) thread #11 - MinaThreadPool" prio=3 tid=0x091c6800 nid=0x34
waiting on condition [0xb5657000]
java.lang.Thread.State: WAITING (parking)
at sun.misc.Unsafe.park(Native Method)
- parking to wait for <0xbb1abfd0> (a
java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
at java.util.concurrent.locks.LockSupport.park(Unknown Source)
at
java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(Unknown
Source)
at java.util.concurrent.LinkedBlockingQueue.take(Unknown Source)
at java.util.concurrent.ThreadPoolExecutor.getTask(Unknown Source)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(Unknown Source)
at java.lang.Thread.run(Unknown Source)
"Camel (mrs) thread #10 - MinaThreadPool" prio=3 tid=0x0990fc00 nid=0x33
waiting on condition [0xb56a8000]
java.lang.Thread.State: WAITING (parking)
at sun.misc.Unsafe.park(Native Method)
- parking to wait for <0xbb1abfd0> (a
java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
at java.util.concurrent.locks.LockSupport.park(Unknown Source)
at
java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(Unknown
Source)
at java.util.concurrent.LinkedBlockingQueue.take(Unknown Source)
at java.util.concurrent.ThreadPoolExecutor.getTask(Unknown Source)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(Unknown Source)
at java.lang.Thread.run(Unknown Source)
"Camel (mrs) thread #9 - MinaThreadPool" prio=3 tid=0x09bba400 nid=0x32
waiting on condition [0xb56f9000]
java.lang.Thread.State: WAITING (parking)
at sun.misc.Unsafe.park(Native Method)
- parking to wait for <0xbb1abfd0> (a
java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
at java.util.concurrent.locks.LockSupport.park(Unknown Source)
at
java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(Unknown
Source)
at java.util.concurrent.LinkedBlockingQueue.take(Unknown Source)
at java.util.concurrent.ThreadPoolExecutor.getTask(Unknown Source)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(Unknown Source)
at java.lang.Thread.run(Unknown Source)
"Camel (mrs) thread #8 - MinaThreadPool" prio=3 tid=0x08467400 nid=0x31
waiting on condition [0xb574a000]
java.lang.Thread.State: WAITING (parking)
at sun.misc.Unsafe.park(Native Method)
- parking to wait for <0xbb1abfd0> (a
java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
at java.util.concurrent.locks.LockSupport.park(Unknown Source)
at
java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(Unknown
Source)
at java.util.concurrent.LinkedBlockingQueue.take(Unknown Source)
at java.util.concurrent.ThreadPoolExecutor.getTask(Unknown Source)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(Unknown Source)
at java.lang.Thread.run(Unknown Source)
"Camel (mrs) thread #7 - MinaThreadPool" prio=3 tid=0x09256400 nid=0x30
waiting on condition [0xb579b000]
java.lang.Thread.State: WAITING (parking)
at sun.misc.Unsafe.park(Native Method)
- parking to wait for <0xbb1abfd0> (a
java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
at java.util.concurrent.locks.LockSupport.park(Unknown Source)
at
java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(Unknown
Source)
at java.util.concurrent.LinkedBlockingQueue.take(Unknown Source)
at java.util.concurrent.ThreadPoolExecutor.getTask(Unknown Source)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(Unknown Source)
at java.lang.Thread.run(Unknown Source)
"Camel (mrs) thread #6 - MinaThreadPool" prio=3 tid=0x09ae9000 nid=0x2f
waiting on condition [0xb57ec000]
java.lang.Thread.State: WAITING (parking)
at sun.misc.Unsafe.park(Native Method)
- parking to wait for <0xbb1abfd0> (a
java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
at java.util.concurrent.locks.LockSupport.park(Unknown Source)
at
java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(Unknown
Source)
at java.util.concurrent.LinkedBlockingQueue.take(Unknown Source)
at java.util.concurrent.ThreadPoolExecutor.getTask(Unknown Source)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(Unknown Source)
at java.lang.Thread.run(Unknown Source)
"Camel (mrs) thread #5 - MinaSocketConnector" prio=3 tid=0x084a9400
nid=0x2e waiting on condition [0xb583d000]
java.lang.Thread.State: WAITING (parking)
at sun.misc.Unsafe.park(Native Method)
- parking to wait for <0xbb1ac2d0> (a
java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
at java.util.concurrent.locks.LockSupport.park(Unknown Source)
at
java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(Unknown
Source)
at java.util.concurrent.LinkedBlockingQueue.take(Unknown Source)
at java.util.concurrent.ThreadPoolExecutor.getTask(Unknown Source)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(Unknown Source)
at java.lang.Thread.run(Unknown Source)
"Camel (mrs) thread #4 - MinaSocketConnector" prio=3 tid=0x09115c00
nid=0x2d waiting on condition [0xb588e000]
java.lang.Thread.State: WAITING (parking)
at sun.misc.Unsafe.park(Native Method)
- parking to wait for <0xbb1ac2d0> (a
java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
at java.util.concurrent.locks.LockSupport.park(Unknown Source)
at
java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(Unknown
Source)
at java.util.concurrent.LinkedBlockingQueue.take(Unknown Source)
at java.util.concurrent.ThreadPoolExecutor.getTask(Unknown Source)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(Unknown Source)
at java.lang.Thread.run(Unknown Source)
"DestroyJavaVM" prio=3 tid=0x08070000 nid=0x2 waiting on condition
[0x00000000]
java.lang.Thread.State: RUNNABLE
"DefaultQuartzScheduler-heartbeat_QuartzSchedulerThread" prio=3
tid=0x09117800 nid=0x2b in Object.wait() [0xb5987000]
java.lang.Thread.State: TIMED_WAITING (on object monitor)
at java.lang.Object.wait(Native Method)
at
org.quartz.core.QuartzSchedulerThread.run(QuartzSchedulerThread.java:296)
- locked <0xbb448e80> (a java.lang.Object)
"DefaultQuartzScheduler-heartbeat_Worker-10" prio=3 tid=0x093c1000 nid=0x2a
in Object.wait() [0xb59d8000]
java.lang.Thread.State: TIMED_WAITING (on object monitor)
at java.lang.Object.wait(Native Method)
at
org.quartz.simpl.SimpleThreadPool$WorkerThread.run(SimpleThreadPool.java:543)
- locked <0xbb4405a8> (a
org.quartz.simpl.SimpleThreadPool$WorkerThread)
"DefaultQuartzScheduler-heartbeat_Worker-9" prio=3 tid=0x093bfc00 nid=0x29
in Object.wait() [0xb5a29000]
java.lang.Thread.State: TIMED_WAITING (on object monitor)
at java.lang.Object.wait(Native Method)
at
org.quartz.simpl.SimpleThreadPool$WorkerThread.run(SimpleThreadPool.java:543)
- locked <0xbb4403c8> (a
org.quartz.simpl.SimpleThreadPool$WorkerThread)
"DefaultQuartzScheduler-heartbeat_Worker-8" prio=3 tid=0x09444c00 nid=0x28
in Object.wait() [0xb5a7a000]
java.lang.Thread.State: TIMED_WAITING (on object monitor)
at java.lang.Object.wait(Native Method)
at
org.quartz.simpl.SimpleThreadPool$WorkerThread.run(SimpleThreadPool.java:543)
- locked <0xbb4401e8> (a
org.quartz.simpl.SimpleThreadPool$WorkerThread)
"DefaultQuartzScheduler-heartbeat_Worker-7" prio=3 tid=0x09443800 nid=0x27
in Object.wait() [0xb5acb000]
java.lang.Thread.State: TIMED_WAITING (on object monitor)
at java.lang.Object.wait(Native Method)
at
org.quartz.simpl.SimpleThreadPool$WorkerThread.run(SimpleThreadPool.java:543)
- locked <0xbb440008> (a
org.quartz.simpl.SimpleThreadPool$WorkerThread)
"DefaultQuartzScheduler-heartbeat_Worker-6" prio=3 tid=0x09442400 nid=0x26
in Object.wait() [0xb5b1c000]
java.lang.Thread.State: TIMED_WAITING (on object monitor)
at java.lang.Object.wait(Native Method)
at
org.quartz.simpl.SimpleThreadPool$WorkerThread.run(SimpleThreadPool.java:543)
- locked <0xbb43fe28> (a
org.quartz.simpl.SimpleThreadPool$WorkerThread)
"DefaultQuartzScheduler-heartbeat_Worker-5" prio=3 tid=0x08ef5c00 nid=0x25
in Object.wait() [0xb5b6d000]
java.lang.Thread.State: TIMED_WAITING (on object monitor)
at java.lang.Object.wait(Native Method)
at
org.quartz.simpl.SimpleThreadPool$WorkerThread.run(SimpleThreadPool.java:543)
- locked <0xbb43fc48> (a
org.quartz.simpl.SimpleThreadPool$WorkerThread)
"DefaultQuartzScheduler-heartbeat_Worker-4" prio=3 tid=0x0953c000 nid=0x24
in Object.wait() [0xb5bbe000]
java.lang.Thread.State: TIMED_WAITING (on object monitor)
at java.lang.Object.wait(Native Method)
at
org.quartz.simpl.SimpleThreadPool$WorkerThread.run(SimpleThreadPool.java:543)
- locked <0xbb43fa68> (a
org.quartz.simpl.SimpleThreadPool$WorkerThread)
"DefaultQuartzScheduler-heartbeat_Worker-3" prio=3 tid=0x09446c00 nid=0x23
in Object.wait() [0xb5c33000]
java.lang.Thread.State: TIMED_WAITING (on object monitor)
at java.lang.Object.wait(Native Method)
at
org.quartz.simpl.SimpleThreadPool$WorkerThread.run(SimpleThreadPool.java:543)
- locked <0xbb43f888> (a
org.quartz.simpl.SimpleThreadPool$WorkerThread)
"DefaultQuartzScheduler-heartbeat_Worker-2" prio=3 tid=0x09a77000 nid=0x22
in Object.wait() [0xb5c84000]
java.lang.Thread.State: TIMED_WAITING (on object monitor)
at java.lang.Object.wait(Native Method)
at
org.quartz.simpl.SimpleThreadPool$WorkerThread.run(SimpleThreadPool.java:543)
- locked <0xbb43f6a8> (a
org.quartz.simpl.SimpleThreadPool$WorkerThread)
"DefaultQuartzScheduler-heartbeat_Worker-1" prio=3 tid=0x098bb000 nid=0x21
in Object.wait() [0xb5cd5000]
java.lang.Thread.State: TIMED_WAITING (on object monitor)
at java.lang.Object.wait(Native Method)
at
org.quartz.simpl.SimpleThreadPool$WorkerThread.run(SimpleThreadPool.java:543)
- locked <0xbb43ed98> (a
org.quartz.simpl.SimpleThreadPool$WorkerThread)
"Camel (mrs) thread #0 - JmsConsumer[ispy.in]" daemon prio=3 tid=0x09a10c00
nid=0x1c in Object.wait() [0xb5e6a000]
java.lang.Thread.State: TIMED_WAITING (on object monitor)
at java.lang.Object.wait(Native Method)
at
org.apache.activemq.SimplePriorityMessageDispatchChannel.dequeue(SimplePriorityMessageDispatchChannel.java:89)
- locked <0xbb4904a0> (a java.lang.Object)
at
org.apache.activemq.ActiveMQMessageConsumer.dequeue(ActiveMQMessageConsumer.java:452)
at
org.apache.activemq.ActiveMQMessageConsumer.receive(ActiveMQMessageConsumer.java:577)
at
org.springframework.jms.listener.AbstractPollingMessageListenerContainer.receiveMessage(AbstractPollingMessageListenerContainer.java:429)
at
org.springframework.jms.listener.AbstractPollingMessageListenerContainer.doReceiveAndExecute(AbstractPollingMessageListenerContainer.java:310)
at
org.springframework.jms.listener.AbstractPollingMessageListenerContainer.receiveAndExecute(AbstractPollingMessageListenerContainer.java:243)
at
org.springframework.jms.listener.DefaultMessageListenerContainer$AsyncMessageListenerInvoker.invokeListener(DefaultMessageListenerContainer.java:1058)
at
org.springframework.jms.listener.DefaultMessageListenerContainer$AsyncMessageListenerInvoker.executeOngoingLoop(DefaultMessageListenerContainer.java:1050)
at
org.springframework.jms.listener.DefaultMessageListenerContainer$AsyncMessageListenerInvoker.run(DefaultMessageListenerContainer.java:947)
at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(Unknown
Source)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(Unknown Source)
at java.lang.Thread.run(Unknown Source)
"PooledByteBufferExpirer-0" daemon prio=3 tid=0x09118400 nid=0x17 waiting
on condition [0xb604d000]
java.lang.Thread.State: TIMED_WAITING (sleeping)
at java.lang.Thread.sleep(Native Method)
at
org.apache.mina.common.PooledByteBufferAllocator$Expirer.run(PooledByteBufferAllocator.java:249)
"HashSessionScavenger-0" daemon prio=3 tid=0x0874fc00 nid=0x15 in
Object.wait() [0xb609e000]
java.lang.Thread.State: TIMED_WAITING (on object monitor)
at java.lang.Object.wait(Native Method)
at java.util.TimerThread.mainLoop(Unknown Source)
- locked <0xbaee27a8> (a java.util.TaskQueue)
at java.util.TimerThread.run(Unknown Source)
"qtp2864404-17" prio=3 tid=0x0831a800 nid=0x14 waiting on condition
[0xb6337000]
java.lang.Thread.State: TIMED_WAITING (parking)
at sun.misc.Unsafe.park(Native Method)
- parking to wait for <0xbae121d8> (a
java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
at java.util.concurrent.locks.LockSupport.parkNanos(Unknown Source)
at
java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(Unknown
Source)
at
org.eclipse.jetty.util.BlockingArrayQueue.poll(BlockingArrayQueue.java:320)
at
org.eclipse.jetty.util.thread.QueuedThreadPool$2.run(QueuedThreadPool.java:480)
at java.lang.Thread.run(Unknown Source)
"qtp2864404-16" prio=3 tid=0x08cc8800 nid=0x13 waiting on condition
[0xb6388000]
java.lang.Thread.State: TIMED_WAITING (parking)
at sun.misc.Unsafe.park(Native Method)
- parking to wait for <0xbae121d8> (a
java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
at java.util.concurrent.locks.LockSupport.parkNanos(Unknown Source)
at
java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(Unknown
Source)
at
org.eclipse.jetty.util.BlockingArrayQueue.poll(BlockingArrayQueue.java:320)
at
org.eclipse.jetty.util.thread.QueuedThreadPool$2.run(QueuedThreadPool.java:480)
at java.lang.Thread.run(Unknown Source)
"qtp2864404-15" prio=3 tid=0x08cc7400 nid=0x12 waiting on condition
[0xb63d9000]
java.lang.Thread.State: TIMED_WAITING (parking)
at sun.misc.Unsafe.park(Native Method)
- parking to wait for <0xbae121d8> (a
java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
at java.util.concurrent.locks.LockSupport.parkNanos(Unknown Source)
at
java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(Unknown
Source)
at
org.eclipse.jetty.util.BlockingArrayQueue.poll(BlockingArrayQueue.java:320)
at
org.eclipse.jetty.util.thread.QueuedThreadPool$2.run(QueuedThreadPool.java:480)
at java.lang.Thread.run(Unknown Source)
"qtp2864404-14" prio=3 tid=0x08cc6c00 nid=0x11 waiting on condition
[0xb642a000]
java.lang.Thread.State: TIMED_WAITING (parking)
at sun.misc.Unsafe.park(Native Method)
- parking to wait for <0xbae121d8> (a
java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
at java.util.concurrent.locks.LockSupport.parkNanos(Unknown Source)
at
java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(Unknown
Source)
at
org.eclipse.jetty.util.BlockingArrayQueue.poll(BlockingArrayQueue.java:320)
at
org.eclipse.jetty.util.thread.QueuedThreadPool$2.run(QueuedThreadPool.java:480)
at java.lang.Thread.run(Unknown Source)
"qtp2864404-13" prio=3 tid=0x08cd9800 nid=0x10 waiting on condition
[0xb647b000]
java.lang.Thread.State: TIMED_WAITING (parking)
at sun.misc.Unsafe.park(Native Method)
- parking to wait for <0xbae121d8> (a
java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
at java.util.concurrent.locks.LockSupport.parkNanos(Unknown Source)
at
java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(Unknown
Source)
at
org.eclipse.jetty.util.BlockingArrayQueue.poll(BlockingArrayQueue.java:320)
at
org.eclipse.jetty.util.thread.QueuedThreadPool$2.run(QueuedThreadPool.java:480)
at java.lang.Thread.run(Unknown Source)
"qtp2864404-12" prio=3 tid=0x08cd9400 nid=0xf runnable [0xb64cc000]
java.lang.Thread.State: RUNNABLE
at sun.nio.ch.ServerSocketChannelImpl.accept0(Native Method)
at sun.nio.ch.ServerSocketChannelImpl.accept(Unknown Source)
- locked <0xbb2c9700> (a java.lang.Object)
at
org.eclipse.jetty.server.nio.SelectChannelConnector$2.run(SelectChannelConnector.java:287)
at
org.eclipse.jetty.util.thread.QueuedThreadPool$2.run(QueuedThreadPool.java:451)
at java.lang.Thread.run(Unknown Source)
"qtp2864404-11" prio=3 tid=0x08982800 nid=0xe waiting on condition
[0xb651d000]
java.lang.Thread.State: TIMED_WAITING (parking)
at sun.misc.Unsafe.park(Native Method)
- parking to wait for <0xbae121d8> (a
java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
at java.util.concurrent.locks.LockSupport.parkNanos(Unknown Source)
at
java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(Unknown
Source)
at
org.eclipse.jetty.util.BlockingArrayQueue.poll(BlockingArrayQueue.java:320)
at
org.eclipse.jetty.util.thread.QueuedThreadPool$2.run(QueuedThreadPool.java:480)
at java.lang.Thread.run(Unknown Source)
"qtp2864404-10 - Acceptor0 DefaultJettyManager$2.0.0.0:9080" prio=3
tid=0x0880d400 nid=0xd runnable [0xb656e000]
java.lang.Thread.State: RUNNABLE
at sun.nio.ch.DevPollArrayWrapper.poll0(Native Method)
at sun.nio.ch.DevPollArrayWrapper.poll(Unknown Source)
at sun.nio.ch.DevPollSelectorImpl.doSelect(Unknown Source)
at sun.nio.ch.SelectorImpl.lockAndDoSelect(Unknown Source)
- locked <0xbb395dc0> (a sun.nio.ch.Util$2)
- locked <0xbb395dd0> (a java.util.Collections$UnmodifiableSet)
- locked <0xbb395d78> (a sun.nio.ch.DevPollSelectorImpl)
at sun.nio.ch.SelectorImpl.select(Unknown Source)
at
org.eclipse.jetty.io.nio.SelectorManager$SelectSet.doSelect(SelectorManager.java:481)
at
org.eclipse.jetty.io.nio.SelectorManager.doSelect(SelectorManager.java:219)
at
org.eclipse.jetty.server.nio.SelectChannelConnector.accept(SelectChannelConnector.java:122)
at
org.eclipse.jetty.server.AbstractConnector$Acceptor.run(AbstractConnector.java:837)
at
org.eclipse.jetty.util.thread.QueuedThreadPool$2.run(QueuedThreadPool.java:451)
at java.lang.Thread.run(Unknown Source)
"Low Memory Detector" daemon prio=3 tid=0x0813c800 nid=0xb runnable
[0x00000000]
java.lang.Thread.State: RUNNABLE
"CompilerThread1" daemon prio=3 tid=0x0813ac00 nid=0xa waiting on condition
[0x00000000]
java.lang.Thread.State: RUNNABLE
"CompilerThread0" daemon prio=3 tid=0x08137800 nid=0x9 waiting on condition
[0x00000000]
java.lang.Thread.State: RUNNABLE
"Signal Dispatcher" daemon prio=3 tid=0x08136400 nid=0x8 waiting on
condition [0x00000000]
java.lang.Thread.State: RUNNABLE
"Finalizer" daemon prio=3 tid=0x08122800 nid=0x7 in Object.wait()
[0xb67e4000]
java.lang.Thread.State: WAITING (on object monitor)
at java.lang.Object.wait(Native Method)
at java.lang.ref.ReferenceQueue.remove(Unknown Source)
- locked <0xbae16668> (a java.lang.ref.ReferenceQueue$Lock)
at java.lang.ref.ReferenceQueue.remove(Unknown Source)
at java.lang.ref.Finalizer$FinalizerThread.run(Unknown Source)
"Reference Handler" daemon prio=3 tid=0x08121400 nid=0x6 in Object.wait()
[0xb6ca7000]
java.lang.Thread.State: WAITING (on object monitor)
at java.lang.Object.wait(Native Method)
at java.lang.Object.wait(Object.java:485)
at java.lang.ref.Reference$ReferenceHandler.run(Unknown Source)
- locked <0xbae16640> (a java.lang.ref.Reference$Lock)
"VM Thread" prio=3 tid=0x0811ec00 nid=0x5 runnable
"GC task thread#0 (ParallelGC)" prio=3 tid=0x08076c00 nid=0x3 runnable
"GC task thread#1 (ParallelGC)" prio=3 tid=0x08078000 nid=0x4 runnable
"VM Periodic Task Thread" prio=3 tid=0x08146800 nid=0xc waiting on
condition
JNI global references: 979
Heap
PSYoungGen total 349248K, used 147749K [0xe58b0000, 0xfae00000,
0xfae00000)
eden space 348992K, 42% used [0xe58b0000,0xee8e95d8,0xfad80000)
from space 256K, 25% used [0xfadc0000,0xfadd0000,0xfae00000)
to space 256K, 0% used [0xfad80000,0xfad80000,0xfadc0000)
PSOldGen total 15936K, used 10389K [0xbae00000, 0xbbd90000,
0xe58b0000)
object space 15936K, 65% used [0xbae00000,0xbb825568,0xbbd90000)
PSPermGen total 23808K, used 23731K [0xb6e00000, 0xb8540000,
0xbae00000)
object space 23808K, 99% used [0xb6e00000,0xb852ce98,0xb8540000)
