iterate actually does a dispatch. when consumers change etc, we again try and dispatch to ensure prefetch buffers are maintained.
Do a svn up to r834579 as I committed a fix to trunk for this today resolving https://issues.apache.org/activemq/browse/AMQ-2481 2009/11/10 afei <afei1...@126.com>: > > in org.apache.activemq.broker.region ,why so many invoke asyncWakeup(), > what is the method:iterate() doing? > > > afei wrote: >> >> in addition,another problem of OOM. >> >> >> >> Gary Tully wrote: >>> >>> fyi: you can disable periodic message expiry processing using a >>> destination policy entry that sets expireMessagesPeriod = 0 >>> >>> 2009/11/9 afei <afei1...@126.com>: >>>> >>>> >>>> when a large of messages in queue,and no consumer or the consumer is >>>> very >>>> slow, the OOM problem occur, because : >>>> in org.apache.activemq.broker.region.Queue,the 588 line is : >>>> doBrowse(true, browsedMessages, this.getMaxExpirePageSize()); >>>> ,transform to : >>>> doBrowse(false, browsedMessages, this.getMaxExpirePageSize()); >>>> is ok. >>>> >>>> >>>> Dejan Bosanac wrote: >>>>> >>>>> Hi Mitch, >>>>> >>>>> yeah, I said in thread I was referring to, that it is working with >>>>> "regular" >>>>> stomp connector. I started investigating AMQ-2440 patch the other day, >>>>> should be have something soon. >>>>> >>>>> Cheers >>>>> -- >>>>> Dejan Bosanac - http://twitter.com/dejanb >>>>> >>>>> Open Source Integration - http://fusesource.com/ >>>>> ActiveMQ in Action - http://www.manning.com/snyder/ >>>>> Blog - http://www.nighttale.net >>>>> >>>>> >>>>> On Wed, Oct 28, 2009 at 6:18 PM, Mitch Granger >>>>> <mitch.gran...@sophos.com>wrote: >>>>> >>>>>> So we turned off stomp+nio and went back to plain old stomp and so far >>>>>> it's >>>>>> working fine. New(IO) isn't always better, I guess :-) >>>>>> >>>>>> Seems like maybe it's this issue -> >>>>>> https://issues.apache.org/activemq/browse/AMQ-2440 >>>>>> >>>>>> >>>>>> afei wrote: >>>>>> >>>>>>> i have same problem >>>>>>> >>>>>>> http://www.nabble.com/file/p26093204/aaaaaa.jpg aaaaaa.jpg >>>>>>> >>>>>>> >>>>>>> themitchy wrote: >>>>>>> >>>>>>>> This is what we've done to tune so far: >>>>>>>> >>>>>>>> - UseDedicatedTaskRunner=false >>>>>>>> - flow control is off >>>>>>>> - stomp transport uses transport.closeAsync=false >>>>>>>> >>>>>>>> I agree that it is because of the high number of open/close >>>>>>>> connections >>>>>>>> from Stomp. When we monitor through JConsole we can see more >>>>>>>> threads >>>>>>>> starting up for each new connection. The problem is that these >>>>>>>> threads >>>>>>>> don't get let go. Even though the stomp clients are disconnecting >>>>>>>> the >>>>>>>> number of threads that get released is less than the number created. >>>>>>>> So >>>>>>>> the thread count goes up and up until it fails. All of the above >>>>>>>> settings/tuning only delay when it will hit the wall. >>>>>>>> >>>>>>>> Dejan Bosanac wrote: >>>>>>>> >>>>>>>>> Hi Mitch, >>>>>>>>> >>>>>>>>> I think the root cause of this problem is that you probably have >>>>>>>>> Stomp >>>>>>>>> clients that open/close connection at a high rate. I simulated this >>>>>>>>> problem >>>>>>>>> on OSX with a StompLoadTest ( >>>>>>>>> >>>>>>>>> http://svn.apache.org/viewvc/activemq/trunk/activemq-core/src/test/java/org/apache/activemq/transport/stomp/StompLoadTest.java?view=log >>>>>>>>> ), >>>>>>>>> while trying to reproduce "too many open files" problem. You can >>>>>>>>> find >>>>>>>>> some >>>>>>>>> of my findings (and workaround) in this thread. >>>>>>>>> >>>>>>>>> >>>>>>>>> http://www.nabble.com/%22too-many-open-files%22-error-with-5.3-and-Stomp-tt25888831.html#a26010080 >>>>>>>>> >>>>>>>>> (BTW. it is producing "too many open files" problem on linux) >>>>>>>>> Basically, >>>>>>>>> the >>>>>>>>> problem with stomp is that every send is done in separate >>>>>>>>> connection >>>>>>>>> and >>>>>>>>> thus considered to be a new producer for every message. So when >>>>>>>>> producer >>>>>>>>> flow control is hit, the producers are piling up and probably not >>>>>>>>> releasing >>>>>>>>> connections. Thus you can observe large number of tcp connections >>>>>>>>> on >>>>>>>>> the >>>>>>>>> system in state TIME_WAIT (and TIME_CLOSE), which causes that the >>>>>>>>> system >>>>>>>>> limit is hit at one point. In the above thread, you can find a >>>>>>>>> workaround >>>>>>>>> that worked for me for that test. I started investigating this more >>>>>>>>> and >>>>>>>>> hopefully I'll have some more findings in the near future. >>>>>>>>> >>>>>>>>> Cheers >>>>>>>>> -- >>>>>>>>> Dejan Bosanac - http://twitter.com/dejanb >>>>>>>>> >>>>>>>>> Open Source Integration - http://fusesource.com/ >>>>>>>>> ActiveMQ in Action - http://www.manning.com/snyder/ >>>>>>>>> Blog - http://www.nighttale.net >>>>>>>>> >>>>>>>>> >>>>>>>>> On Tue, Oct 27, 2009 at 1:07 AM, Mitch Granger >>>>>>>>> <mitch.gran...@sophos.com>wrote: >>>>>>>>> >>>>>>>>> Update: We've [nearly] proven that this only happens with AMQ >>>>>>>>> running >>>>>>>>>> on >>>>>>>>>> openVZ. What exactly is causing it, we're still not sure. After >>>>>>>>>> memoryUsage is met, the number of threads skyrockets until we get >>>>>>>>>> OutOfMemoryError. >>>>>>>>>> >>>>>>>>>> It works just fine on regular hardware; We're going to try VMWare >>>>>>>>>> tomorrow. >>>>>>>>>> >>>>>>>>>> One thing really worth mentioning is that by using the fileCursor >>>>>>>>>> we >>>>>>>>>> actually started seeing it use the Temp Store. When reading about >>>>>>>>>> systemUsage it is NOT intuitive that the Temp Store does not come >>>>>>>>>> into >>>>>>>>>> play >>>>>>>>>> with the default cursor. Anyone keeping a significant volume of >>>>>>>>>> messages on >>>>>>>>>> their queues should be well served by changing the cursor. >>>>>>>>>> >>>>>>>>>> >>>>>>>>>> Mitch Granger wrote: >>>>>>>>>> >>>>>>>>>> Config is attached. We have also tried the >>>>>>>>>> activemq-scalability.xml >>>>>>>>>>> with >>>>>>>>>>> the only change being adding a stomp connector. >>>>>>>>>>> >>>>>>>>>>> Once we hit the memoryUsage limit we can [sometimes] connect new >>>>>>>>>>> consumers >>>>>>>>>>> but nothing comes back after we send the SUBSCRIBE frame. >>>>>>>>>>> >>>>>>>>>>> I expect sending to fail when we hit this limit but if we can't >>>>>>>>>>> subscribe >>>>>>>>>>> there's no chance of recovering from this state. >>>>>>>>>>> >>>>>>>>>>> Rob Davies wrote: >>>>>>>>>>> >>>>>>>>>>> On 26 Oct 2009, at 17:38, themitchy wrote: >>>>>>>>>>>> >>>>>>>>>>>> We're using only persistent messages and heap size is set to >>>>>>>>>>>> 2GB >>>>>>>>>>>> yet >>>>>>>>>>>> we >>>>>>>>>>>> >>>>>>>>>>>>> hit >>>>>>>>>>>>> the memoryUsage limit quite quickly (system usage config >>>>>>>>>>>>> below). >>>>>>>>>>>>> This >>>>>>>>>>>>> is >>>>>>>>>>>>> followed by "java.lang.OutOfMemoryError: unable to create new >>>>>>>>>>>>> native >>>>>>>>>>>>> thread" >>>>>>>>>>>>> as the process quickly reaches the 2GB of heap we gave it. How >>>>>>>>>>>>> are >>>>>>>>>>>>> we >>>>>>>>>>>>> getting to that point with the memoryUsage limit set far below >>>>>>>>>>>>> it? >>>>>>>>>>>>> >>>>>>>>>>>>> Is there no way to get AMQ to gracefully limit it's memory >>>>>>>>>>>>> usage? >>>>>>>>>>>>> >>>>>>>>>>>>> <systemUsage> >>>>>>>>>>>>> <systemUsage> >>>>>>>>>>>>> <memoryUsage> >>>>>>>>>>>>> <memoryUsage limit="256 mb"/> >>>>>>>>>>>>> </memoryUsage> >>>>>>>>>>>>> <storeUsage> >>>>>>>>>>>>> <storeUsage limit="60 gb" name="foo"/> >>>>>>>>>>>>> </storeUsage> >>>>>>>>>>>>> <tempUsage> >>>>>>>>>>>>> <tempUsage limit="60 gb"/> >>>>>>>>>>>>> </tempUsage> >>>>>>>>>>>>> </systemUsage> >>>>>>>>>>>>> </systemUsage> >>>>>>>>>>>>> >>>>>>>>>>>>> -- >>>>>>>>>>>>> View this message in context: >>>>>>>>>>>>> http://www.nabble.com/Out-of-Memory-on-5.3-tp26064098p26064098.html >>>>>>>>>>>>> Sent from the ActiveMQ - User mailing list archive at >>>>>>>>>>>>> Nabble.com. >>>>>>>>>>>>> >>>>>>>>>>>>> >>>>>>>>>>>>> Can you send the rest of your config ? >>>>>>>>>>>> >>>>>>>>>>>> Rob Davies >>>>>>>>>>>> http://twitter.com/rajdavies >>>>>>>>>>>> I work here: http://fusesource.com >>>>>>>>>>>> My Blog: http://rajdavies.blogspot.com/ >>>>>>>>>>>> I'm writing this: http://www.manning.com/snyder/ >>>>>>>>>>>> >>>>>>>>>>>> >>>>>>>>>>>> >>>>>>>>>>>> >>>>>>>>>>>> >>>>>>>>>>>> >>>>>>>>>>>> >>>>>>>> >>>>>>> >>>>> >>>>> >>>>> ----- >>>>> Dejan Bosanac >>>>> >>>>> Open Source Integration - http://fusesource.com/ >>>>> ActiveMQ in Action - http://www.manning.com/snyder/ >>>>> Blog - http://www.nighttale.net >>>>> >>>> http://old.nabble.com/file/p26264779/dump.jpg >>>> -- >>>> View this message in context: >>>> http://old.nabble.com/Out-of-Memory-on-5.3-tp26064098p26264779.html >>>> Sent from the ActiveMQ - User mailing list archive at Nabble.com. >>>> >>>> >>> >>> >>> >>> -- >>> http://blog.garytully.com >>> >>> Open Source Integration >>> http://fusesource.com >>> >>> >> http://old.nabble.com/file/p26278228/oom.jpg >> > > -- > View this message in context: > http://old.nabble.com/Out-of-Memory-on-5.3-tp26064098p26279671.html > Sent from the ActiveMQ - User mailing list archive at Nabble.com. > > -- http://blog.garytully.com Open Source Integration http://fusesource.com
