I checked logs. there are bunch of Info messages about conection taking too long to shutdown.
2014-04-15 10:51:00,668 | INFO | The connection to 'tcp:// 10.190.133.15:37168' is taking a long time to shutdown. | org.apache.activemq.broker.TransportConnection | ActiveMQ Transport: tcp:///10.190.133.15:55356@61616 2014-04-15 10:51:00,669 | INFO | The connection to 'tcp:// 10.190.133.14:58333' is taking a long time to shutdown. | org.apache.activemq.broker.TransportConnection | ActiveMQ Transport: tcp:///10.190.133.14:48314@61616 2014-04-15 10:51:00,740 | INFO | The connection to 'tcp:// 10.190.133.15:54436' is taking a long time to shutdown. | org.apache.activemq.broker.TransportConnection | ActiveMQ Transport: tcp:///10.190.133.15:55568@61616 2014-04-15 10:51:00,741 | INFO | The connection to 'tcp:// 10.190.133.15:37172' is taking a long time to shutdown. | org.apache.activemq.broker.TransportConnection | ActiveMQ Transport: tcp:///10.190.133.15:55575@61616 2014-04-15 10:51:00,989 | INFO | The connection to 'tcp:// 10.190.133.14:58360' is taking a long time to shutdown. | org.apache.activemq.broker.TransportConnection | ActiveMQ Transport: tcp:///10.190.133.14:48072@61616 2014-04-15 10:51:00,989 | INFO | The connection to 'tcp:// 10.190.133.14:58370' is taking a long time to shutdown. | org.apache.activemq.broker.TransportConnection | ActiveMQ Transport: tcp:///10.190.133.14:48290@61616 2014-04-15 10:51:00,994 | INFO | The connection to 'tcp:// 10.190.133.14:47258' is taking a long time to shutdown. | org.apache.activemq.broker.TransportConnection | ActiveMQ Transport: tcp:///10.190.133.14:48296@61616 2014-04-15 10:51:01,755 | INFO | The connection to 'tcp:// 10.190.133.14:47191' is taking a long time to shutdown. | org.apache.activemq.broker.TransportConnection | ActiveMQ Transport: tcp:///10.190.133.14:47985@61616 2014-04-15 10:51:01,935 | INFO | The connection to 'tcp:// 10.190.133.14:46963' is taking a long time to shutdown. | org.apache.activemq.broker.TransportConnection | ActiveMQ Transport: tcp:///10.190.133.14:48308@61616 2014-04-15 10:51:02,362 | INFO | The connection to 'tcp:// 10.190.133.15:54359' is taking a long time to shutdown. | org.apache.activemq.broker.TransportConnection | ActiveMQ Transport: tcp:///10.190.133.15:55261@61616 I also checked logs around time when stop processing messages. I saw some warning messages. 2014-04-14 18:08:35,782 | WARN | Transport Connection to: tcp://10.184.132.34:56916 failed: java.net.SocketException: Connection reset | org.apache.activemq.broker.TransportConnection.Transport | ActiveMQ Transport: tcp:///10.184.132.34:56916@61616 2014-04-14 18:52:54,576 | WARN | Transport Connection to: tcp://10.190.225.39:48556 failed: java.io.EOFException | org.apache.activemq.broker.TransportConnection.Transport | ActiveMQ Transport: tcp:///10.190.225.39:48556@61616 2014-04-14 19:43:35,398 | WARN | Transport Connection to: tcp://10.190.225.39:49153 failed: java.io.EOFException | org.apache.activemq.broker.TransportConnection.Transport | ActiveMQ Transport: tcp:///10.190.225.39:49153@61616 2014-04-14 19:43:46,813 | WARN | Transport Connection to: tcp://10.190.133.15:37178 failed: java.net.SocketException: Broken pipe | org.apache.activemq.broker.TransportConnection.Transport | ActiveMQ InactivityMonitor Worker 2014-04-14 19:51:36,954 | WARN | Transport Connection to: tcp://10.190.225.56:51240 failed: java.io.EOFException | org.apache.activemq.broker.TransportConnection.Transport | ActiveMQ Transport: tcp:///10.190.225.56:51240@61616 2014-04-14 20:00:27,651 | WARN | Transport Connection to: tcp://10.190.225.72:53818 failed: java.io.EOFException | org.apache.activemq.broker.TransportConnection.Transport | ActiveMQ Transport: tcp:///10.190.225.72:53818@61616 2014-04-14 20:00:28,096 | WARN | Transport Connection to: tcp://10.190.225.72:53676 failed: java.io.EOFException | org.apache.activemq.broker.TransportConnection.Transport | ActiveMQ Transport: tcp:///10.190.225.72:53676@61616 2014-04-14 20:00:28,097 | WARN | Transport Connection to: tcp://10.190.225.72:53780 failed: java.io.EOFException | org.apache.activemq.broker.TransportConnection.Transport | ActiveMQ Transport: tcp:///10.190.225.72:53780@61616 2014-04-14 20:00:28,097 | WARN | Transport Connection to: tcp://10.190.225.72:53718 failed: java.io.EOFException | org.apache.activemq.broker.TransportConnection.Transport | ActiveMQ Transport: tcp:///10.190.225.72:53718@61616 2014-04-14 20:00:28,098 | WARN | Transport Connection to: tcp://10.190.225.72:53766 failed: java.io.EOFException | org.apache.activemq.broker.TransportConnection.Transport | ActiveMQ Transport: tcp:///10.190.225.72:53766@61616 We have zabbix monitor for activemq. I didnt see anything alarming for memory usage or store usage. Enqueue and dequeue count were almost same most of time. Total message count per minute was almost zero all the time. I found this similar post and there was bug related to issue. but it was fixed in 5.6 release. Does it mean it was not fixed or reintroduced due to other code change. Post: http://activemq.2283324.n4.nabble.com/Connection-taking-long-time-to-shutdown-td3717825.html Bug: https://issues.apache.org/jira/browse/AMQ-3435 any suggestions. thanks, chirag On Tue, Apr 15, 2014 at 9:06 PM, artnaseef <a...@artnaseef.com> wrote: > The most common cause of message flow stopping is slow consumption leading > to > exhaustion of the broker's resources. > > Things to check: > - Broker memory usage > - Broker store usage > - ActiveMQ log file (check for producer flow control or resource usage > messages) > > BTW, it is critical with any ActiveMQ installation to ensure consumption >= > production always. Short bursts do happen and can be handled, but > ultimately, you really never want to be in a situation in which consumption > is less than production for a long time. And, when it is known that a > burst > of production will outpace consumption, it's important to understand the > numbers and sizes of messages that will be in the backlog to ensure the > broker's memory and store settings can handle it. > > > > > -- > View this message in context: > http://activemq.2283324.n4.nabble.com/Activemq-froze-and-msgs-dont-get-processed-tp4680313p4680318.html > Sent from the ActiveMQ - User mailing list archive at Nabble.com. >
