Hi guys,
did some initial testing on this (will commit my test soon, so we can have
it for further testings).
I wasn't able to reproduce the exact error you notice, but was getting a
"unable to create new native thread" from a stomp broker under load. The
test simulates the behavior where producer/consumer open/close conenctions
on each operation.
What I noticed is a lot of sockets in TIME_WAIT (and some in CLOSE_WAIT)
state. After some digging I found that some configuration settings can
improve things
1. Use closeAsync transport option
<transportConnector name="stomp" uri="stomp://
0.0.0.0:61612?transport.closeAsync=false"/>
2. Remove small memoryLimit from destinationPolicy
<destinationPolicy>
<policyMap>
<policyEntries>
<policyEntry topic=">" producerFlowControl="true">
<pendingSubscriberPolicy>
<vmCursor />
</pendingSubscriberPolicy>
</policyEntry>
<policyEntry queue=">" producerFlowControl="true">
<pendingQueuePolicy>
<vmQueueCursor/>
</pendingQueuePolicy>
</policyEntry>
</policyEntries>
</policyMap>
</destinationPolicy>
Since all messages are sent through new connections they are all considered
to be new producers to the broker, and once the flow control is hit,
producers are grouping and probably taking all available connections. You
might try turning off producer flow control completely to see if it works
for you. Found this tip in http://issues.apache.org/activemq/browse/AMQ-1739
3. You can also try some of the scaling techniques, described here
http://activemq.apache.org/how-do-i-configure-10s-of-1000s-of-queues-in-a-single-broker-.html
I turned off dedicated task runner and gave the broker some more memory,
like this
ACTIVEMQ_OPTS="-Xmx1024M -Dorg.apache.activemq.UseDedicatedTaskRunner=false
-Djava.util.logging.config.file=logging.properties"
With all these options, the load test over the standard stomp connector ran
well for quite some time (I still observed some problems with stomp+nio
protocol).
Since there are a few issues regarding stomp and connections
http://issues.apache.org/activemq/browse/AMQ-1739
http://issues.apache.org/activemq/browse/AMQ-1873
https://issues.apache.org/activemq/browse/AMQ-2440
will take more look into it and perhaps when we have it sorted, we should
create activemq-stomp.xml tuned configuration for Stomp usage.
In the meantime, it would be great if you'd test this configuration in your
environment and report if there were any improvements (sample config is
attached)
Cheers
--
Dejan Bosanac - http://twitter.com/dejanb
Open Source Integration - http://fusesource.com/
ActiveMQ in Action - http://www.manning.com/snyder/
Blog - http://www.nighttale.net
On Thu, Oct 22, 2009 at 12:11 PM, DavidLevy <dvid.l...@gmail.com> wrote:
>
> Hi
>
> - it's web, so I can't really tell ...
> I would say 50-100 STOMP PHP producers/consumers
> + 2 REST python producers/consumers
> In 5.3 i was using stomp at first, and then stomp+nio but I think both had
> the issue.
>
> - most of them yes, because it's new PHP scripts
>
> PS : the server is a 64bit server. It might be a problem ?
>
>
>
> Dejan Bosanac wrote:
> >
> > Hi David,
> >
> > ok, can you at least provide more data on:
> >
> > - how many producers/consumers are you using
> > - are you opening a new connection for every message send/receive
> >
> > Also, are you using regular stomp or stomp+nio transports?
> >
> > I'm working on creating a java test that can simulate stomp load
> > environment
> > with many producers/consumers opening connections.
> >
> > Cheers
> > --
> > Dejan Bosanac - http://twitter.com/dejanb
> >
> > Open Source Integration - http://fusesource.com/
> > ActiveMQ in Action - http://www.manning.com/snyder/
> > Blog - http://www.nighttale.net
> >
> >
> > On Wed, Oct 21, 2009 at 4:03 PM, DavidLevy <dvid.l...@gmail.com> wrote:
> >
> >>
> >> Thanks Dejan,
> >>
> >> This happen in my production environment which is made of a dozen
> server.
> >> I don't believe it would be easy to reproduce by myself in a smaller
> >> environment.
> >>
> >> I noticed that it works better with v5.2. With 5.3, it's not stable for
> >> more
> >> than a few hours. 5.2 can last a few days before it crashes.
> >>
> >> I was also thinking about using REST calls instead of STOMP for some PHP
> >> scripts ... It should help if the STOMP leak is my issue for real ...
> >>
> >>
> >>
> >>
> >>
> >> Dejan Bosanac wrote:
> >> >
> >> > I plan to look at these connection closing issues with stomp in coming
> >> > days.
> >> > Any chance you can create a reproducible test case (in any language)?
> >> >
> >> > Cheers
> >> > --
> >> > Dejan Bosanac - http://twitter.com/dejanb
> >> >
> >> > Open Source Integration - http://fusesource.com/
> >> > ActiveMQ in Action - http://www.manning.com/snyder/
> >> > Blog - http://www.nighttale.net
> >> >
> >> >
> >> > On Wed, Oct 14, 2009 at 12:46 PM, DavidLevy <dvid.l...@gmail.com>
> >> wrote:
> >> >
> >> >>
> >> >> Hi
> >> >>
> >> >> I had the issue with 5.2 and now that I installed 5.3 last night, it
> >> >> still
> >> >> the same, or even worse.
> >> >> It basically looks like this issue :
> >> >> http://issues.apache.org/activemq/browse/AMQ-1873
> >> >>
> >> >> After a short time (1 hour or so) the broker is filled with errors
> >> like
> >> >> 'too
> >> >> many open files' and I have to restart activemq.
> >> >>
> >> >>
> >> >> I don't believe it's a performance issue because
> >> >> - the host server is strong
> >> >> - there are less than 10 queues
> >> >> - there are less than 10000 messages
> >> >>
> >> >> I assume it has to do also with Stomp connections not being released.
> >> >> I am using both PHP and Python clients. I believe the issue is
> related
> >> to
> >> >> PHP clients that may be more error prone if the Apache threads die
> >> >> improperly ...
> >> >>
> >> >> I ll attach my current config file.
> >> >>
> >> >> Any help is welcome !
> >> >>
> >> >> Thanks :)
> >> >>
> >> >> http://www.nabble.com/file/p25888831/activemq.xml activemq.xml
> >> >> --
> >> >> View this message in context:
> >> >>
> >>
> http://www.nabble.com/%22too-many-open-files%22-error-with-5.3-and-Stomp-tp25888831p25888831.html
> >> >> Sent from the ActiveMQ - User mailing list archive at Nabble.com.
> >> >>
> >> >>
> >> >
> >> >
> >> > -----
> >> > Dejan Bosanac
> >> >
> >> > Open Source Integration - http://fusesource.com/
> >> > ActiveMQ in Action - http://www.manning.com/snyder/
> >> > Blog - http://www.nighttale.net
> >> >
> >>
> >> --
> >> View this message in context:
> >>
> http://www.nabble.com/%22too-many-open-files%22-error-with-5.3-and-Stomp-tp25888831p25993154.html
> >> Sent from the ActiveMQ - User mailing list archive at Nabble.com.
> >>
> >>
> >
> >
> > -----
> > Dejan Bosanac
> >
> > Open Source Integration - http://fusesource.com/
> > ActiveMQ in Action - http://www.manning.com/snyder/
> > Blog - http://www.nighttale.net
> >
>
> --
> View this message in context:
> http://www.nabble.com/%22too-many-open-files%22-error-with-5.3-and-Stomp-tp25888831p26007460.html
> Sent from the ActiveMQ - User mailing list archive at Nabble.com.
>
>
<!--
Licensed to the Apache Software Foundation (ASF) under one or more
contributor license agreements. See the NOTICE file distributed with
this work for additional information regarding copyright ownership.
The ASF licenses this file to You under the Apache License, Version 2.0
(the "License"); you may not use this file except in compliance with
the License. You may obtain a copy of the License at
http://www.apache.org/licenses/LICENSE-2.0
Unless required by applicable law or agreed to in writing, software
distributed under the License is distributed on an "AS IS" BASIS,
WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
See the License for the specific language governing permissions and
limitations under the License.
-->
<beans
xmlns="http://www.springframework.org/schema/beans";
xmlns:amq="http://activemq.apache.org/schema/core";
xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance";
xsi:schemaLocation="http://www.springframework.org/schema/beans http://www.springframework.org/schema/beans/spring-beans-2.0.xsd
http://activemq.apache.org/schema/core http://activemq.apache.org/schema/core/activemq-core.xsd";>
<!-- Allows us to use system properties as variables in this configuration file -->
<bean class="org.springframework.beans.factory.config.PropertyPlaceholderConfigurer">
<property name="locations">
<value>file:${activemq.base}/conf/credentials.properties</value>
</property>
</bean>
<!--
The <broker> element is used to configure the ActiveMQ broker.
-->
<broker xmlns="http://activemq.apache.org/schema/core"; brokerName="localhost" dataDirectory="${activemq.base}/data">
<!--
The managementContext is used to configure how ActiveMQ is exposed in
JMX. By default, ActiveMQ uses the MBean server that is started by
the JVM. For more information, see:
http://activemq.apache.org/jmx.html
-->
<managementContext>
<managementContext createConnector="false"/>
</managementContext>
<!--
Configure message persistence for the broker. The default persistence
mechanism is the KahaDB store (identified by the kahaDB tag).
For more information, see:
http://activemq.apache.org/persistence.html
-->
<persistenceAdapter>
<kahaDB directory="${activemq.base}/data/kahadb"/>
</persistenceAdapter>
<!--
For better performances use VM cursor and small memory limit.
For more information, see:
http://activemq.apache.org/message-cursors.html
Also, if your producer is "hanging", it's probably due to producer flow control.
For more information, see:
http://activemq.apache.org/producer-flow-control.html
-->
<destinationPolicy>
<policyMap>
<policyEntries>
<policyEntry topic=">" producerFlowControl="true">
<pendingSubscriberPolicy>
<vmCursor />
</pendingSubscriberPolicy>
</policyEntry>
<policyEntry queue=">" producerFlowControl="true">
<pendingQueuePolicy>
<vmQueueCursor/>
</pendingQueuePolicy>
</policyEntry>
</policyEntries>
</policyMap>
</destinationPolicy>
<!--
The systemUsage controls the maximum amount of space the broker will
use before slowing down producers. For more information, see:
http://activemq.apache.org/producer-flow-control.html
<systemUsage>
<systemUsage>
<memoryUsage>
<memoryUsage limit="20 mb"/>
</memoryUsage>
<storeUsage>
<storeUsage limit="1 gb" name="foo"/>
</storeUsage>
<tempUsage>
<tempUsage limit="100 mb"/>
</tempUsage>
</systemUsage>
</systemUsage>
-->
<!--
The transport connectors expose ActiveMQ over a given protocol to
clients and other brokers. For more information, see:
http://activemq.apache.org/configuring-transports.html
-->
<transportConnectors>
<transportConnector name="openwire" uri="tcp://0.0.0.0:61616"/>
<transportConnector name="stomp" uri="stomp://0.0.0.0:61612?transport.closeAsync=false"/>
<transportConnector name="stomp+nio" uri="stomp+nio://0.0.0.0:61613?transport.closeAsync=false"/>
</transportConnectors>
</broker>
<!--
Uncomment to enable Camel
Take a look at activemq-camel.xml for more details
<import resource="camel.xml"/>
-->
<!--
Enable web consoles, REST and Ajax APIs and demos
Take a look at activemq-jetty.xml for more details
-->
<import resource="jetty.xml"/>
</beans>
