I can reproduce this with the non-embedded, out-of-the-box broker. I also think it's an under-the-hood problem.
Until we get a fix, we've set pending message queue limits on our busy topics. So, for example, the broker will only queue up 500 messages on a particular topic per consumer. This will prevent things from getting too out of hand on a single client crash, but is not the solution to the problem. -Kirby -----Original Message----- From: Will van der Leij [mailto:[EMAIL PROTECTED] Sent: Monday, December 08, 2008 12:57 AM To: users@activemq.apache.org Subject: Re: Broker not releasing memory I use an embedded broker too and have gone through all the possible iterations of config options for threading, acknowledgements and message storage. Alas, an apparent "memory leak" still persists with slow consumers that close/die with messages still in-flight/pending. I've had several other people report the same problem with different clients leading me to believe that there must be something odd under the hood. Rick Blair wrote: > > Hi Will, > > We use an embedded broker Here is the url. > > tcp://{0}:{1}?jms.useAsyncSend=false&jms.optimizeAcknowledge=false&jms.prefe > tchPolicy.all=1000"; > > The problem started when optimizeAcknowledge=true was used. Also I cannt > get old messages in the topics to timeout. Some do, most don¹t. > > > Hope this helps. > > Rick > -- > Tìoraidh! > > Rick Blair > Associate Technical Fellow > Boeing Phantom Works > Network Systems Technology > Information Management Program > M/S: 42-50 > Voice: (206) 544-1610 > > > >> From: Will van der Leij <[EMAIL PROTECTED]> >> Reply-To: <users@activemq.apache.org> >> Date: Wed, 3 Dec 2008 23:32:31 -0800 (PST) >> To: <users@activemq.apache.org> >> Subject: Re: Broker not releasing memory >> >> >> Hi Rick, >> As I understand optimizeAcknowledge is disabled by default. >> Unfortunately, >> setting it explicitely has not had an affect on our memory usage. >> Could you perhaps briefly describe the rest of your configuration? I'm >> presuming that it is a combination of options that would do the trick. >> >> Regards, >> Will >> >> >> >> Rick Blair wrote: >>> >>> Hi Kirby, >>> >>> We have seen exactly the same behaviour. As a work around we set >>> optimizeAcknowdlege to false. I did not run down the root cause, >>> however. >>> >>> -- >>> Cheers! >>> >>> Rick Blair >>> >>> >>>> From: "Frugia, Kirby A" <[EMAIL PROTECTED]> >>>> Reply-To: <users@activemq.apache.org> >>>> Date: Wed, 3 Dec 2008 15:43:30 +0000 >>>> To: "users@activemq.apache.org" <users@activemq.apache.org> >>>> Conversation: Broker not releasing memory >>>> Subject: RE: Broker not releasing memory >>>> >>>> This is fairly easy to reproduce if you create one app which sends >>>> messages >>>> quickly and another which consumes the messages and sleeps before >>>> ack'ing. >>>> Once you see broker memory usage going up (through activemq-admin), >>>> kill >>>> the >>>> consumer application. Then run activemq-admin again some time later >>>> and >>>> you'll see the same memorypercentusage. >>>> >>>> Or, let the consumer run the broker out of memory and you'll see the >>>> producer >>>> get blocked. Kill both apps and you'll never see memory go back down >>>> again >>>> and you'll never be able to send messages again. >>>> >>>> Thanks, >>>> Kirby >>>> >>>> -----Original Message----- >>>> From: Will van der Leij [mailto:[EMAIL PROTECTED] >>>> Sent: Wednesday, December 03, 2008 12:17 AM >>>> To: users@activemq.apache.org >>>> Subject: RE: Broker not releasing memory >>>> >>>> >>>> Well, I'm reticent to suggest something going on under the hood without >>>> taking a good look at it, however, if there is something wrong in the >>>> broker >>>> then I presume it has to do with a "sweeper"-like thread that >>>> periodically >>>> cleans up queues and topics that are left pending and consumerless >>>> (similar >>>> to how sweeper threads would clean up expired messages etc. >>>> >>>> Then again, we are the only two that I know of describing this >>>> behaviour. >>>> This suggests that either there is something bizarrely odd about our >>>> configurations or we are, indeed, correct :) >>>> >>>> >>>> >>>> Frugia, Kirby A wrote: >>>>> >>>>> Yes, this is exactly what we are seeing. Any ideas? >>>>> >>>>> -----Original Message----- >>>>> From: Will van der Leij [mailto:[EMAIL PROTECTED] >>>>> Sent: Thursday, November 27, 2008 11:42 PM >>>>> To: users@activemq.apache.org >>>>> Subject: Re: Broker not releasing memory >>>>> >>>>> >>>>> We see similar behaviour in a slightly different setup so I'd like to >>>>> elaborate a bit in the hope of finding out if this is either intended >>>>> behaviour or a memory leak. >>>>> >>>>> Essentially, with a single or multiple producers and a single or >>>>> multiple >>>>> consumers on a single topic: >>>>> - We send messages faster than the consumer can read asynchronously >>>>> - When the send queue is larger than the consumer's prefetch buffer >>>>> then >>>>> we correctly see messages filling up the broker memory >>>>> (memoryPercentUSage in JMX) >>>>> - Killing the consumer (cleanly through a close or not-cleanly with a >>>>> terminate signal) >>>>> results in the memoryPercentUsage not being released >>>>> - even though there are no more consumers on the Topic >>>>> - this memory is never released until the broker is restarted >>>>> >>>>> I've tried it with persistence on and off, all the retroactive >>>>> recovery >>>>> options on and off, flow control on and off, caching on and off, >>>>> asynchronous dispatching on and off, noLocal flag on and off etc. >>>>> >>>>> This is a very typical case of a consumer dying with inflight messages >>>>> on >>>>> the broker. Is the behaviour of the messages not being cleaned up or >>>>> recovered intended (i.e. they are there for a reason) or is there >>>>> soething >>>>> wrnog with the setup. >>>>> >>>>> Many thanks >>>>> Will van der Leij >>>>> >>>>> >>>>> >>>>> Frugia, Kirby A wrote: >>>>>> >>>>>> Hi All, >>>>>> >>>>>> Sorry I dual-posted in the dev list. I think I sent to the wrong >>>>>> one... >>>>>> >>>>>> We are seeing an issue with our broker not releasing memory on >>>>>> topics. >>>>>> >>>>>> Setup: >>>>>> * Active MQ 5.1 (out of the box) >>>>>> * Persistent messages sent by publishers >>>>>> * Non-durable topics >>>>>> >>>>>> We were seeing an issue in production with slow consumers causing the >>>>>> broker to run out of memory. >>>>>> >>>>>> We wrote a test app, which has one publisher and one subscriber to >>>>>> the >>>>>> same topic. The publisher sends messages on this topic frequently. >>>>>> The >>>>>> subscriber, upon receiving a message, goes to sleep for 10 minutes. >>>>>> >>>>>> Very quickly, the broker runs out of memory. When this happens, our >>>>>> publishers can no longer send messages; the send is blocked. This is >>>>>> expected. However, if we kill our application (which cleanly >>>>>> disconnects >>>>>> from the broker), the broker's memory usage (MemoryPercentUsage) does >>>>>> not >>>>>> go back down. Also, any new apps that startup will have their >>>>>> publishers >>>>>> blocked. >>>>>> >>>>>> Shouldn't the broker release the memory associated with these >>>>>> messages? >>>>>> >>>>>> Thanks, >>>>>> Kirby >>>>>> >>>>>> >>>>>> >>>>> >>>>> -- >>>>> View this message in context: >>>>> http://www.nabble.com/Broker-not-releasing-memory-tp20662078p20730116.html >>>>> Sent from the ActiveMQ - User mailing list archive at Nabble.com. >>>>> >>>>> >>>>> >>>> >>>> -- >>>> View this message in context: >>>> http://www.nabble.com/Broker-not-releasing-memory-tp20662078p20808585.html >>>> Sent from the ActiveMQ - User mailing list archive at Nabble.com. >>>> >>> >>> >>> >> >> -- >> View this message in context: >> http://www.nabble.com/Broker-not-releasing-memory-tp20662078p20828589.html >> Sent from the ActiveMQ - User mailing list archive at Nabble.com. >> > > > -- View this message in context: http://www.nabble.com/Broker-not-releasing-memory-tp20662078p20890719.html Sent from the ActiveMQ - User mailing list archive at Nabble.com.
