Hi List,
I just restarted a activemq 5.3-instance which was causing 1000% cpu
(the machine has 8 cores, so all where fully working) and a load of 1035.
I'm pretty confident that this load/cpu-usage was caused by using
stomp+nio rather than 'stomp' (whether it was just nio or not, I don't
know).
Our usage-pattern is that we have a few queues with a single consumer
each. The producers are php-web-processes that make a connection using
stomp, send a few messages to some of the queues and disconnect again
(which is why we have the transport.closeAsync=false). The heaviest two
queues receive about 30-50 messages/second during peak hours (and thus,
ActiveMQ receives about 30-50 connections/second).
We also have a few topics that get messages quite infrequently, from
similar stomp-producers, but those are consumed with java-based
applications.
The load/cpu-usage apparently has gradually increased, as can be seen in
these two graphs:
Cpu-usage
http://tweakers.net/stats/?Action=Generator&Mode=Serverstats&Time=1263919627&Dagen=1&StatsServer=Argus&colServers=CPUUsage
Load average
http://tweakers.net/stats/?Action=Generator&Mode=Serverstats&Time=1263919627&Dagen=1&StatsServer=Argus&colServers=LoadAvg
The increase in load has gone steadily in just a few hours up untill the
moment where it took the php-processes more than 13 seconds to actually
connect, send two messages and disconnect (rather than a few milliseconds).
Our activemq.xml is attached, although I already replaced stomp+nio with
normal stomp. Our ACTIVEM_OPTS is "-Xms2048M -Xmx2048M
-XX:+UseParallelOldGC -server
-Dorg.apache.activemq.UseDedicatedTaskRunner=true" (i.e. the default
with adjusted memory and GC)
The increasing load hasn't appeared yet with the adjusted config that
uses non-nio connections.
Unfortunately, this is a production server so I can't run tests very
easily. Nor do I have a problem with not running nio, so this is mainly
a mail to let you know of this problem, rather than me having a big
issue with activemq right now :)
Best regards,
Arjen
<!--
Licensed to the Apache Software Foundation (ASF) under one or more
contributor license agreements. See the NOTICE file distributed with
this work for additional information regarding copyright ownership.
The ASF licenses this file to You under the Apache License, Version 2.0
(the "License"); you may not use this file except in compliance with
the License. You may obtain a copy of the License at
http://www.apache.org/licenses/LICENSE-2.0
Unless required by applicable law or agreed to in writing, software
distributed under the License is distributed on an "AS IS" BASIS,
WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
See the License for the specific language governing permissions and
limitations under the License.
-->
<beans
xmlns="http://www.springframework.org/schema/beans";
xmlns:amq="http://activemq.apache.org/schema/core";
xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance";
xsi:schemaLocation="http://www.springframework.org/schema/beans http://www.springframework.org/schema/beans/spring-beans-2.0.xsd
http://activemq.apache.org/schema/core http://activemq.apache.org/schema/core/activemq-core.xsd";>
<!-- Allows us to use system properties as variables in this configuration file -->
<bean class="org.springframework.beans.factory.config.PropertyPlaceholderConfigurer">
<property name="locations">
<value>file:${activemq.base}/conf/credentials.properties</value>
</property>
</bean>
<!--
The <broker> element is used to configure the ActiveMQ broker.
-->
<broker xmlns="http://activemq.apache.org/schema/core"; brokerName="localhost" dataDirectory="${activemq.base}/data">
<!--
The managementContext is used to configure how ActiveMQ is exposed in
JMX. By default, ActiveMQ uses the MBean server that is started by
the JVM. For more information, see:
http://activemq.apache.org/jmx.html
-->
<managementContext>
<managementContext createConnector="false"/>
</managementContext>
<!--
Configure message persistence for the broker. The default persistence
mechanism is the KahaDB store (identified by the kahaDB tag).
For more information, see:
http://activemq.apache.org/persistence.html
-->
<persistenceAdapter>
<kahaDB directory="${activemq.base}/data/kahadb"/>
</persistenceAdapter>
<!--
For better performances use VM cursor and small memory limit.
For more information, see:
http://activemq.apache.org/message-cursors.html
Also, if your producer is "hanging", it's probably due to producer flow control.
For more information, see:
http://activemq.apache.org/producer-flow-control.html
-->
<!-- <destinationPolicy>
<policyMap>
<policyEntries>
<policyEntry topic=">" producerFlowControl="true" memoryLimit="1mb">
<pendingSubscriberPolicy>
<vmCursor />
</pendingSubscriberPolicy>
</policyEntry>
<policyEntry queue=">" producerFlowControl="true" memoryLimit="1mb">
<!- - Use VM cursor for better latency
For more information, see:
http://activemq.apache.org/message-cursors.html
<pendingQueuePolicy>
<vmQueueCursor/>
</pendingQueuePolicy>
- ->
<!- - </policyEntry>
</policyEntries>
</policyMap>
</destinationPolicy> -->
<!--
The systemUsage controls the maximum amount of space the broker will
use before slowing down producers. For more information, see:
http://activemq.apache.org/producer-flow-control.html
<systemUsage>
<systemUsage>
<memoryUsage>
<memoryUsage limit="20 mb"/>
</memoryUsage>
<storeUsage>
<storeUsage limit="1 gb" name="foo"/>
</storeUsage>
<tempUsage>
<tempUsage limit="100 mb"/>
</tempUsage>
</systemUsage>
</systemUsage>
-->
<!--
The transport connectors expose ActiveMQ over a given protocol to
clients and other brokers. For more information, see:
http://activemq.apache.org/configuring-transports.html
-->
<transportConnectors>
<transportConnector name="openwire" uri="tcp://0.0.0.0:61616?transport.closeAsync=false"/>
<transportConnector name="stomp" uri="stomp://0.0.0.0:61613?transport.closeAsync=false" />
</transportConnectors>
</broker>
<!--
Uncomment to enable Camel
Take a look at activemq-camel.xml for more details
<import resource="camel.xml"/>
-->
<!--
Enable web consoles, REST and Ajax APIs and demos
Take a look at activemq-jetty.xml for more details
-->
<import resource="jetty.xml"/>
</beans>
