We did find this in the logs. activemq.log:2010-09-08 10:55:02,555 | INFO | Usage Manager Memory Limit is full. Producer (ID:isp01-55443-1283902621643-0:0:1:1) stopped to prevent flooding queue://mpqueue. See http://activemq.apache.org/producer-flow-control.html for more info (blocking for: 1s) | org.apache.activemq.broker.re gion.Queue | ActiveMQ Transport: tcp:///10.13.85.14:52682
We have since increased the memory limit and we'll see if this happens again. If so I'll get a full thread dump. Thanks, -M@ On Sep 10, 2010, at 12:51 AM, Dejan Bosanac wrote: > Can you turn on debug logging and post logs and thread dumps when the > broker hangs. Also, do you see messages in logs about producer flow > control being used? > > > Cheers > -- > Dejan Bosanac - http://twitter.com/dejanb > > Open Source Integration - http://fusesource.com/ > ActiveMQ in Action - http://www.manning.com/snyder/ > Blog - http://www.nighttale.net > > > > On Thu, Sep 9, 2010 at 11:33 PM, Matthew Hixson <mhix...@gmail.com> wrote: >> We have been running 5.4.0 in production for a few weeks and it can easily >> handle tens of thousands of messages a day, but then for no apparent reason >> it will cease accepting new messages from our producers. The consumers will >> drain the queue empty, but then nothing new gets accepted into Active MQ. >> Restarting Active MQ and then restarting all the producers and consumers >> allows things to proceed as normal. >> Below is our activemq.xml. >> Thanks for any help. >> -M@ >> >> <!-- >> Licensed to the Apache Software Foundation (ASF) under one or more >> contributor license agreements. See the NOTICE file distributed with >> this work for additional information regarding copyright ownership. >> The ASF licenses this file to You under the Apache License, Version 2.0 >> (the "License"); you may not use this file except in compliance with >> the License. You may obtain a copy of the License at >> >> http://www.apache.org/licenses/LICENSE-2.0 >> >> Unless required by applicable law or agreed to in writing, software >> distributed under the License is distributed on an "AS IS" BASIS, >> WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. >> See the License for the specific language governing permissions and >> limitations under the License. >> --> >> <beans >> xmlns="http://www.springframework.org/schema/beans"; >> xmlns:amq="http://activemq.apache.org/schema/core"; >> xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"; >> xsi:schemaLocation="http://www.springframework.org/schema/beans >> http://www.springframework.org/schema/beans/spring-beans-2.0.xsd >> http://activemq.apache.org/schema/core >> http://activemq.apache.org/schema/core/activemq-core.xsd";> >> >> <!-- Allows us to use system properties as variables in this >> configuration file --> >> <bean >> class="org.springframework.beans.factory.config.PropertyPlaceholderConfigurer"> >> <property name="locations"> >> <value>file:${activemq.base}/conf/credentials.properties</value> >> </property> >> </bean> >> >> <!-- >> The <broker> element is used to configure the ActiveMQ broker. >> --> >> <broker xmlns="http://activemq.apache.org/schema/core"; >> brokerName="localhost" dataDirectory="${activemq.base}/data" >> destroyApplicationContextOnStop="true"> >> >> <!-- >> For better performances use VM cursor and small >> memory limit. >> For more information, see: >> >> http://activemq.apache.org/message-cursors.html >> >> Also, if your producer is "hanging", it's probably due to >> producer flow control. >> For more information, see: >> http://activemq.apache.org/producer-flow-control.html >> --> >> >> <destinationPolicy> >> <policyMap> >> <policyEntries> >> <policyEntry topic=">" producerFlowControl="false" >> memoryLimit="100mb"> >> <pendingSubscriberPolicy> >> <vmCursor /> >> </pendingSubscriberPolicy> >> </policyEntry> >> <policyEntry queue=">" producerFlowControl="true" >> memoryLimit="1mb"> >> <!-- Use VM cursor for better latency >> For more information, see: >> >> http://activemq.apache.org/message-cursors.html >> >> <pendingQueuePolicy> >> <vmQueueCursor/> >> </pendingQueuePolicy> >> --> >> </policyEntry> >> </policyEntries> >> </policyMap> >> </destinationPolicy> >> >> >> <!-- >> The managementContext is used to configure how ActiveMQ is >> exposed in >> JMX. By default, ActiveMQ uses the MBean server that is started by >> the JVM. For more information, see: >> >> http://activemq.apache.org/jmx.html >> --> >> <managementContext> >> <managementContext createConnector="false"/> >> </managementContext> >> >> <!-- >> Configure message persistence for the broker. The default >> persistence >> mechanism is the KahaDB store (identified by the kahaDB tag). >> For more information, see: >> >> http://activemq.apache.org/persistence.html >> --> >> <persistenceAdapter> >> <kahaDB directory="${activemq.base}/data/kahadb"/> >> </persistenceAdapter> >> >> >> <!-- >> The systemUsage controls the maximum amount of space the broker >> will >> use before slowing down producers. For more information, see: >> >> http://activemq.apache.org/producer-flow-control.html >> >> <systemUsage> >> <systemUsage> >> <memoryUsage> >> <memoryUsage limit="100 mb"/> >> </memoryUsage> >> <storeUsage> >> <storeUsage limit="10 gb"/> >> </storeUsage> >> <tempUsage> >> <tempUsage limit="100 mb"/> >> </tempUsage> >> </systemUsage> >> </systemUsage> >> --> >> >> <!-- >> The transport connectors expose ActiveMQ over a given protocol to >> clients and other brokers. For more information, see: >> >> http://activemq.apache.org/configuring-transports.html >> --> >> <transportConnectors> >> <transportConnector name="openwire" uri="tcp://0.0.0.0:61616"/> >> </transportConnectors> >> >> </broker> >> >> <!-- >> Uncomment to enable Camel >> Take a look at activemq-camel.xml for more details >> >> <import resource="camel.xml"/> >> --> >> >> <!-- >> Enable web consoles, REST and Ajax APIs and demos >> Take a look at activemq-jetty.xml for more details >> --> >> <import resource="jetty.xml"/> >> >> </beans>
