So we turned off stomp+nio and went back to plain old stomp and so far
it's working fine. New(IO) isn't always better, I guess :-)
Seems like maybe it's this issue ->
https://issues.apache.org/activemq/browse/AMQ-2440
afei wrote:
i have same problem
http://www.nabble.com/file/p26093204/aaaaaa.jpg aaaaaa.jpg
themitchy wrote:
This is what we've done to tune so far:
- UseDedicatedTaskRunner=false
- flow control is off
- stomp transport uses transport.closeAsync=false
I agree that it is because of the high number of open/close connections
from Stomp. When we monitor through JConsole we can see more threads
starting up for each new connection. The problem is that these threads
don't get let go. Even though the stomp clients are disconnecting the
number of threads that get released is less than the number created. So
the thread count goes up and up until it fails. All of the above
settings/tuning only delay when it will hit the wall.
Dejan Bosanac wrote:
Hi Mitch,
I think the root cause of this problem is that you probably have Stomp
clients that open/close connection at a high rate. I simulated this
problem
on OSX with a StompLoadTest (
http://svn.apache.org/viewvc/activemq/trunk/activemq-core/src/test/java/org/apache/activemq/transport/stomp/StompLoadTest.java?view=log),
while trying to reproduce "too many open files" problem. You can find
some
of my findings (and workaround) in this thread.
http://www.nabble.com/%22too-many-open-files%22-error-with-5.3-and-Stomp-tt25888831.html#a26010080
(BTW. it is producing "too many open files" problem on linux) Basically,
the
problem with stomp is that every send is done in separate connection and
thus considered to be a new producer for every message. So when producer
flow control is hit, the producers are piling up and probably not
releasing
connections. Thus you can observe large number of tcp connections on the
system in state TIME_WAIT (and TIME_CLOSE), which causes that the system
limit is hit at one point. In the above thread, you can find a workaround
that worked for me for that test. I started investigating this more and
hopefully I'll have some more findings in the near future.
Cheers
--
Dejan Bosanac - http://twitter.com/dejanb
Open Source Integration - http://fusesource.com/
ActiveMQ in Action - http://www.manning.com/snyder/
Blog - http://www.nighttale.net
On Tue, Oct 27, 2009 at 1:07 AM, Mitch Granger
<mitch.gran...@sophos.com>wrote:
Update: We've [nearly] proven that this only happens with AMQ running on
openVZ. What exactly is causing it, we're still not sure. After
memoryUsage is met, the number of threads skyrockets until we get
OutOfMemoryError.
It works just fine on regular hardware; We're going to try VMWare
tomorrow.
One thing really worth mentioning is that by using the fileCursor we
actually started seeing it use the Temp Store. When reading about
systemUsage it is NOT intuitive that the Temp Store does not come into
play
with the default cursor. Anyone keeping a significant volume of
messages on
their queues should be well served by changing the cursor.
Mitch Granger wrote:
Config is attached. We have also tried the activemq-scalability.xml
with
the only change being adding a stomp connector.
Once we hit the memoryUsage limit we can [sometimes] connect new
consumers
but nothing comes back after we send the SUBSCRIBE frame.
I expect sending to fail when we hit this limit but if we can't
subscribe
there's no chance of recovering from this state.
Rob Davies wrote:
On 26 Oct 2009, at 17:38, themitchy wrote:
We're using only persistent messages and heap size is set to 2GB yet
we
hit
the memoryUsage limit quite quickly (system usage config below).
This
is
followed by "java.lang.OutOfMemoryError: unable to create new native
thread"
as the process quickly reaches the 2GB of heap we gave it. How are
we
getting to that point with the memoryUsage limit set far below it?
Is there no way to get AMQ to gracefully limit it's memory usage?
<systemUsage>
<systemUsage>
<memoryUsage>
<memoryUsage limit="256 mb"/>
</memoryUsage>
<storeUsage>
<storeUsage limit="60 gb" name="foo"/>
</storeUsage>
<tempUsage>
<tempUsage limit="60 gb"/>
</tempUsage>
</systemUsage>
</systemUsage>
--
View this message in context:
http://www.nabble.com/Out-of-Memory-on-5.3-tp26064098p26064098.html
Sent from the ActiveMQ - User mailing list archive at Nabble.com.
Can you send the rest of your config ?
Rob Davies
http://twitter.com/rajdavies
I work here: http://fusesource.com
My Blog: http://rajdavies.blogspot.com/
I'm writing this: http://www.manning.com/snyder/
