fyi: you can disable periodic message expiry processing using a destination policy entry that sets expireMessagesPeriod = 0
2009/11/9 afei <afei1...@126.com>: > > > when a large of messages in queue,and no consumer or the consumer is very > slow, the OOM problem occur, because : > in org.apache.activemq.broker.region.Queue,the 588 line is : > doBrowse(true, browsedMessages, this.getMaxExpirePageSize()); > ,transform to : > doBrowse(false, browsedMessages, this.getMaxExpirePageSize()); > is ok. > > > Dejan Bosanac wrote: >> >> Hi Mitch, >> >> yeah, I said in thread I was referring to, that it is working with >> "regular" >> stomp connector. I started investigating AMQ-2440 patch the other day, >> should be have something soon. >> >> Cheers >> -- >> Dejan Bosanac - http://twitter.com/dejanb >> >> Open Source Integration - http://fusesource.com/ >> ActiveMQ in Action - http://www.manning.com/snyder/ >> Blog - http://www.nighttale.net >> >> >> On Wed, Oct 28, 2009 at 6:18 PM, Mitch Granger >> <mitch.gran...@sophos.com>wrote: >> >>> So we turned off stomp+nio and went back to plain old stomp and so far >>> it's >>> working fine. New(IO) isn't always better, I guess :-) >>> >>> Seems like maybe it's this issue -> >>> https://issues.apache.org/activemq/browse/AMQ-2440 >>> >>> >>> afei wrote: >>> >>>> i have same problem >>>> >>>> http://www.nabble.com/file/p26093204/aaaaaa.jpg aaaaaa.jpg >>>> >>>> >>>> themitchy wrote: >>>> >>>>> This is what we've done to tune so far: >>>>> >>>>> - UseDedicatedTaskRunner=false >>>>> - flow control is off >>>>> - stomp transport uses transport.closeAsync=false >>>>> >>>>> I agree that it is because of the high number of open/close connections >>>>> from Stomp. When we monitor through JConsole we can see more threads >>>>> starting up for each new connection. The problem is that these threads >>>>> don't get let go. Even though the stomp clients are disconnecting the >>>>> number of threads that get released is less than the number created. >>>>> So >>>>> the thread count goes up and up until it fails. All of the above >>>>> settings/tuning only delay when it will hit the wall. >>>>> >>>>> Dejan Bosanac wrote: >>>>> >>>>>> Hi Mitch, >>>>>> >>>>>> I think the root cause of this problem is that you probably have Stomp >>>>>> clients that open/close connection at a high rate. I simulated this >>>>>> problem >>>>>> on OSX with a StompLoadTest ( >>>>>> >>>>>> http://svn.apache.org/viewvc/activemq/trunk/activemq-core/src/test/java/org/apache/activemq/transport/stomp/StompLoadTest.java?view=log >>>>>> ), >>>>>> while trying to reproduce "too many open files" problem. You can find >>>>>> some >>>>>> of my findings (and workaround) in this thread. >>>>>> >>>>>> >>>>>> http://www.nabble.com/%22too-many-open-files%22-error-with-5.3-and-Stomp-tt25888831.html#a26010080 >>>>>> >>>>>> (BTW. it is producing "too many open files" problem on linux) >>>>>> Basically, >>>>>> the >>>>>> problem with stomp is that every send is done in separate connection >>>>>> and >>>>>> thus considered to be a new producer for every message. So when >>>>>> producer >>>>>> flow control is hit, the producers are piling up and probably not >>>>>> releasing >>>>>> connections. Thus you can observe large number of tcp connections on >>>>>> the >>>>>> system in state TIME_WAIT (and TIME_CLOSE), which causes that the >>>>>> system >>>>>> limit is hit at one point. In the above thread, you can find a >>>>>> workaround >>>>>> that worked for me for that test. I started investigating this more >>>>>> and >>>>>> hopefully I'll have some more findings in the near future. >>>>>> >>>>>> Cheers >>>>>> -- >>>>>> Dejan Bosanac - http://twitter.com/dejanb >>>>>> >>>>>> Open Source Integration - http://fusesource.com/ >>>>>> ActiveMQ in Action - http://www.manning.com/snyder/ >>>>>> Blog - http://www.nighttale.net >>>>>> >>>>>> >>>>>> On Tue, Oct 27, 2009 at 1:07 AM, Mitch Granger >>>>>> <mitch.gran...@sophos.com>wrote: >>>>>> >>>>>> Update: We've [nearly] proven that this only happens with AMQ running >>>>>>> on >>>>>>> openVZ. What exactly is causing it, we're still not sure. After >>>>>>> memoryUsage is met, the number of threads skyrockets until we get >>>>>>> OutOfMemoryError. >>>>>>> >>>>>>> It works just fine on regular hardware; We're going to try VMWare >>>>>>> tomorrow. >>>>>>> >>>>>>> One thing really worth mentioning is that by using the fileCursor we >>>>>>> actually started seeing it use the Temp Store. When reading about >>>>>>> systemUsage it is NOT intuitive that the Temp Store does not come >>>>>>> into >>>>>>> play >>>>>>> with the default cursor. Anyone keeping a significant volume of >>>>>>> messages on >>>>>>> their queues should be well served by changing the cursor. >>>>>>> >>>>>>> >>>>>>> Mitch Granger wrote: >>>>>>> >>>>>>> Config is attached. We have also tried the activemq-scalability.xml >>>>>>>> with >>>>>>>> the only change being adding a stomp connector. >>>>>>>> >>>>>>>> Once we hit the memoryUsage limit we can [sometimes] connect new >>>>>>>> consumers >>>>>>>> but nothing comes back after we send the SUBSCRIBE frame. >>>>>>>> >>>>>>>> I expect sending to fail when we hit this limit but if we can't >>>>>>>> subscribe >>>>>>>> there's no chance of recovering from this state. >>>>>>>> >>>>>>>> Rob Davies wrote: >>>>>>>> >>>>>>>> On 26 Oct 2009, at 17:38, themitchy wrote: >>>>>>>>> >>>>>>>>> We're using only persistent messages and heap size is set to 2GB >>>>>>>>> yet >>>>>>>>> we >>>>>>>>> >>>>>>>>>> hit >>>>>>>>>> the memoryUsage limit quite quickly (system usage config below). >>>>>>>>>> This >>>>>>>>>> is >>>>>>>>>> followed by "java.lang.OutOfMemoryError: unable to create new >>>>>>>>>> native >>>>>>>>>> thread" >>>>>>>>>> as the process quickly reaches the 2GB of heap we gave it. How >>>>>>>>>> are >>>>>>>>>> we >>>>>>>>>> getting to that point with the memoryUsage limit set far below it? >>>>>>>>>> >>>>>>>>>> Is there no way to get AMQ to gracefully limit it's memory usage? >>>>>>>>>> >>>>>>>>>> <systemUsage> >>>>>>>>>> <systemUsage> >>>>>>>>>> <memoryUsage> >>>>>>>>>> <memoryUsage limit="256 mb"/> >>>>>>>>>> </memoryUsage> >>>>>>>>>> <storeUsage> >>>>>>>>>> <storeUsage limit="60 gb" name="foo"/> >>>>>>>>>> </storeUsage> >>>>>>>>>> <tempUsage> >>>>>>>>>> <tempUsage limit="60 gb"/> >>>>>>>>>> </tempUsage> >>>>>>>>>> </systemUsage> >>>>>>>>>> </systemUsage> >>>>>>>>>> >>>>>>>>>> -- >>>>>>>>>> View this message in context: >>>>>>>>>> http://www.nabble.com/Out-of-Memory-on-5.3-tp26064098p26064098.html >>>>>>>>>> Sent from the ActiveMQ - User mailing list archive at Nabble.com. >>>>>>>>>> >>>>>>>>>> >>>>>>>>>> Can you send the rest of your config ? >>>>>>>>> >>>>>>>>> Rob Davies >>>>>>>>> http://twitter.com/rajdavies >>>>>>>>> I work here: http://fusesource.com >>>>>>>>> My Blog: http://rajdavies.blogspot.com/ >>>>>>>>> I'm writing this: http://www.manning.com/snyder/ >>>>>>>>> >>>>>>>>> >>>>>>>>> >>>>>>>>> >>>>>>>>> >>>>>>>>> >>>>>>>>> >>>>> >>>> >> >> >> ----- >> Dejan Bosanac >> >> Open Source Integration - http://fusesource.com/ >> ActiveMQ in Action - http://www.manning.com/snyder/ >> Blog - http://www.nighttale.net >> > http://old.nabble.com/file/p26264779/dump.jpg > -- > View this message in context: > http://old.nabble.com/Out-of-Memory-on-5.3-tp26064098p26264779.html > Sent from the ActiveMQ - User mailing list archive at Nabble.com. > > -- http://blog.garytully.com Open Source Integration http://fusesource.com
