Hi,
We appear to be facing a deadlock during queue message delivery. See
the following stack dump:
"ActiveMQ Transport: tcp:///10.10.10.30:40919" daemon prio=10
tid=0x00002aab002a3800 nid=0x281e in Object.wait()
[0x0000000057b77000]
java.lang.Thread.State: TIMED_WAITING (on object monitor)
at java.lang.Object.wait(Native Method)
at org.apache.activemq.usage.Usage.waitForSpace(Usage.java:99)
- locked <0x00002aaab37c9e68> (a java.lang.Object)
at org.apache.activemq.broker.region.Queue.doMessageSend(Queue.java:496)
- locked <0x00002aaab3a8cb48> (a java.lang.Object)
at org.apache.activemq.broker.region.Queue.send(Queue.java:480)
[...snip...]
and several (15) of these:
"ActiveMQ Transport: tcp:///10.10.10.31:55890" daemon prio=10
tid=0x00002aab00fbd800 nid=0x27f8 waiting for monitor entry
[0x000000004f26a000]
java.lang.Thread.State: BLOCKED (on object monitor)
at
org.apache.activemq.broker.region.Queue.doMessageSend(Queue.java:489)
- waiting to
lock <0x00002aaab3a8cb48> (a java.lang.Object)
at org.apache.activemq.broker.region.Queue.send(Queue.java:480)
We're using 5.3 (via fusesource) with the Kaha persistence adapter.
Full broker config can be seen here -
http://pastie.org/581968
It would seem from the stack trace above and from looking at the
source, that it thinks the store usage is greater than 100% (just a
guess) and is waiting for it to fall back to normal levels, however
the entire data folder was only 630mb at the time of the deadlock. On
restarting the server (had to kill -9 due to unresponsive threads) the
data folder was no longer readable so I had to remove it (have a
backup) in order to get the server started again.
Any ideas as to what's going on here?
regards,
chetan
