when a large of messages in queue,and no consumer or the consumer is very slow, the OOM problem occur, because : in org.apache.activemq.broker.region.Queue,the 588 line is : doBrowse(true, browsedMessages, this.getMaxExpirePageSize()); ,transform to : doBrowse(false, browsedMessages, this.getMaxExpirePageSize()); is ok.
Dejan Bosanac wrote: > > Hi Mitch, > > yeah, I said in thread I was referring to, that it is working with > "regular" > stomp connector. I started investigating AMQ-2440 patch the other day, > should be have something soon. > > Cheers > -- > Dejan Bosanac - http://twitter.com/dejanb > > Open Source Integration - http://fusesource.com/ > ActiveMQ in Action - http://www.manning.com/snyder/ > Blog - http://www.nighttale.net > > > On Wed, Oct 28, 2009 at 6:18 PM, Mitch Granger > <mitch.gran...@sophos.com>wrote: > >> So we turned off stomp+nio and went back to plain old stomp and so far >> it's >> working fine. New(IO) isn't always better, I guess :-) >> >> Seems like maybe it's this issue -> >> https://issues.apache.org/activemq/browse/AMQ-2440 >> >> >> afei wrote: >> >>> i have same problem >>> >>> http://www.nabble.com/file/p26093204/aaaaaa.jpg aaaaaa.jpg >>> >>> >>> themitchy wrote: >>> >>>> This is what we've done to tune so far: >>>> >>>> - UseDedicatedTaskRunner=false >>>> - flow control is off >>>> - stomp transport uses transport.closeAsync=false >>>> >>>> I agree that it is because of the high number of open/close connections >>>> from Stomp. When we monitor through JConsole we can see more threads >>>> starting up for each new connection. The problem is that these threads >>>> don't get let go. Even though the stomp clients are disconnecting the >>>> number of threads that get released is less than the number created. >>>> So >>>> the thread count goes up and up until it fails. All of the above >>>> settings/tuning only delay when it will hit the wall. >>>> >>>> Dejan Bosanac wrote: >>>> >>>>> Hi Mitch, >>>>> >>>>> I think the root cause of this problem is that you probably have Stomp >>>>> clients that open/close connection at a high rate. I simulated this >>>>> problem >>>>> on OSX with a StompLoadTest ( >>>>> >>>>> http://svn.apache.org/viewvc/activemq/trunk/activemq-core/src/test/java/org/apache/activemq/transport/stomp/StompLoadTest.java?view=log >>>>> ), >>>>> while trying to reproduce "too many open files" problem. You can find >>>>> some >>>>> of my findings (and workaround) in this thread. >>>>> >>>>> >>>>> http://www.nabble.com/%22too-many-open-files%22-error-with-5.3-and-Stomp-tt25888831.html#a26010080 >>>>> >>>>> (BTW. it is producing "too many open files" problem on linux) >>>>> Basically, >>>>> the >>>>> problem with stomp is that every send is done in separate connection >>>>> and >>>>> thus considered to be a new producer for every message. So when >>>>> producer >>>>> flow control is hit, the producers are piling up and probably not >>>>> releasing >>>>> connections. Thus you can observe large number of tcp connections on >>>>> the >>>>> system in state TIME_WAIT (and TIME_CLOSE), which causes that the >>>>> system >>>>> limit is hit at one point. In the above thread, you can find a >>>>> workaround >>>>> that worked for me for that test. I started investigating this more >>>>> and >>>>> hopefully I'll have some more findings in the near future. >>>>> >>>>> Cheers >>>>> -- >>>>> Dejan Bosanac - http://twitter.com/dejanb >>>>> >>>>> Open Source Integration - http://fusesource.com/ >>>>> ActiveMQ in Action - http://www.manning.com/snyder/ >>>>> Blog - http://www.nighttale.net >>>>> >>>>> >>>>> On Tue, Oct 27, 2009 at 1:07 AM, Mitch Granger >>>>> <mitch.gran...@sophos.com>wrote: >>>>> >>>>> Update: We've [nearly] proven that this only happens with AMQ running >>>>>> on >>>>>> openVZ. What exactly is causing it, we're still not sure. After >>>>>> memoryUsage is met, the number of threads skyrockets until we get >>>>>> OutOfMemoryError. >>>>>> >>>>>> It works just fine on regular hardware; We're going to try VMWare >>>>>> tomorrow. >>>>>> >>>>>> One thing really worth mentioning is that by using the fileCursor we >>>>>> actually started seeing it use the Temp Store. When reading about >>>>>> systemUsage it is NOT intuitive that the Temp Store does not come >>>>>> into >>>>>> play >>>>>> with the default cursor. Anyone keeping a significant volume of >>>>>> messages on >>>>>> their queues should be well served by changing the cursor. >>>>>> >>>>>> >>>>>> Mitch Granger wrote: >>>>>> >>>>>> Config is attached. We have also tried the activemq-scalability.xml >>>>>>> with >>>>>>> the only change being adding a stomp connector. >>>>>>> >>>>>>> Once we hit the memoryUsage limit we can [sometimes] connect new >>>>>>> consumers >>>>>>> but nothing comes back after we send the SUBSCRIBE frame. >>>>>>> >>>>>>> I expect sending to fail when we hit this limit but if we can't >>>>>>> subscribe >>>>>>> there's no chance of recovering from this state. >>>>>>> >>>>>>> Rob Davies wrote: >>>>>>> >>>>>>> On 26 Oct 2009, at 17:38, themitchy wrote: >>>>>>>> >>>>>>>> We're using only persistent messages and heap size is set to 2GB >>>>>>>> yet >>>>>>>> we >>>>>>>> >>>>>>>>> hit >>>>>>>>> the memoryUsage limit quite quickly (system usage config below). >>>>>>>>> This >>>>>>>>> is >>>>>>>>> followed by "java.lang.OutOfMemoryError: unable to create new >>>>>>>>> native >>>>>>>>> thread" >>>>>>>>> as the process quickly reaches the 2GB of heap we gave it. How >>>>>>>>> are >>>>>>>>> we >>>>>>>>> getting to that point with the memoryUsage limit set far below it? >>>>>>>>> >>>>>>>>> Is there no way to get AMQ to gracefully limit it's memory usage? >>>>>>>>> >>>>>>>>> <systemUsage> >>>>>>>>> <systemUsage> >>>>>>>>> <memoryUsage> >>>>>>>>> <memoryUsage limit="256 mb"/> >>>>>>>>> </memoryUsage> >>>>>>>>> <storeUsage> >>>>>>>>> <storeUsage limit="60 gb" name="foo"/> >>>>>>>>> </storeUsage> >>>>>>>>> <tempUsage> >>>>>>>>> <tempUsage limit="60 gb"/> >>>>>>>>> </tempUsage> >>>>>>>>> </systemUsage> >>>>>>>>> </systemUsage> >>>>>>>>> >>>>>>>>> -- >>>>>>>>> View this message in context: >>>>>>>>> http://www.nabble.com/Out-of-Memory-on-5.3-tp26064098p26064098.html >>>>>>>>> Sent from the ActiveMQ - User mailing list archive at Nabble.com. >>>>>>>>> >>>>>>>>> >>>>>>>>> Can you send the rest of your config ? >>>>>>>> >>>>>>>> Rob Davies >>>>>>>> http://twitter.com/rajdavies >>>>>>>> I work here: http://fusesource.com >>>>>>>> My Blog: http://rajdavies.blogspot.com/ >>>>>>>> I'm writing this: http://www.manning.com/snyder/ >>>>>>>> >>>>>>>> >>>>>>>> >>>>>>>> >>>>>>>> >>>>>>>> >>>>>>>> >>>> >>> > > > ----- > Dejan Bosanac > > Open Source Integration - http://fusesource.com/ > ActiveMQ in Action - http://www.manning.com/snyder/ > Blog - http://www.nighttale.net > http://old.nabble.com/file/p26264779/dump.jpg -- View this message in context: http://old.nabble.com/Out-of-Memory-on-5.3-tp26064098p26264779.html Sent from the ActiveMQ - User mailing list archive at Nabble.com.
