Re: Broker Leak

2012-12-14 Thread Christian Posta
Jerry, Thanks again for demonstrating this. It's been fixed in the trunk. https://issues.apache.org/jira/browse/AMQ-4222 Thanks, Christian On Thu, Dec 13, 2012 at 11:17 AM, Christian Posta wrote: > Awesome, thanks for coming back and explaining what happened there. I was > super curious abou

Re: Broker Leak

2012-12-13 Thread Christian Posta
Awesome, thanks for coming back and explaining what happened there. I was super curious about this one. I still think there's room for a unit test to document this and show that the reference in the ProducerBrokerExchange still holds onto the "region" destinations. This reference could be cleared a

Re: Broker Leak

2012-12-13 Thread Jerry Cwiklik
Christian, after much pain and suffering I finally figured out what is going on. Our system is quite complicated and involves many producers that send large messages (600K-1.5M) to a relatively few multi-threaded consumers (services) which run "forever". The producers are transient and can be kille

Re: Broker Leak

2012-12-12 Thread Christian Posta
Bad choice of words.. not "destined" but could run out of memory if consumers cannot keep up with producers On Wed, Dec 12, 2012 at 5:51 PM, Christian Posta wrote: > So i guess in general i would think that a configuration with no PFC, vm > cursors, and no persistence is destined to run out of m

Re: Broker Leak

2012-12-12 Thread Christian Posta
So i guess in general i would think that a configuration with no PFC, vm cursors, and no persistence is destined to run out of memory. I wonder if what's happening in your case is these temporary queues are getting filled with messages that aren't being consumed and then when the reference hangs a

Re: Broker Leak

2012-12-12 Thread Jerry Cwiklik
Christian, thanks. The most recent experiment was run with a much smaller broker, Xmx=360M. I am trying to reproduce the problem I am seeing in a larger system which is not easy to debug. I can't share the large heap dump as it may contain proprietary data. But thanks for the offer to help examine

Re: Broker Leak

2012-12-11 Thread Christian Posta
No, that's not expected behavior. I ran your tests as you described and I do see memory being consumed and churned, but it all eventually gets collected by the GC. Is your heap the 10G heap you mentioned originally? If you can get me the heapdump, I'll gladly take a look for you. Let me know and I

Re: Broker Leak

2012-12-11 Thread Jerry Cwiklik
Thanks, your reply seems consistent with what I am seeing in the heap dump (see heapAnalyzer screenshot attached to the initial post). I am still unclear how are the dead messages cleaned up (if ever). I ran a new scenario where I had 4 SpringConsumerWithReply processes and a single ProducerWithRep

Re: Broker Leak

2012-12-11 Thread Jerry Cwiklik
Thanks, yes this is true that there is an open connection to a temp queue from the the SpringConsumerWithReply service. This is probably a common scenario where one deploys a service process which services requests from clients which can disappear at any time. Cleaning up unused/dead connections wo

Re: Broker Leak

2012-12-11 Thread Christian Posta
So here's what's happening... There are actually three (3) connections being created: 1.. by your ProducerWithConsumer.java 1.. by the Spring DMLC 1.. by your explicit code in SpringConsumerWithReply (which you use to produce to the temp destination). So the connection that I saw laying around w

Re: Broker Leak

2012-12-11 Thread Jerry Cwiklik
Our broker has been configured with We are experiencing the problem with the above setting. Also, the broker occasionally logs the following: WARN Transport -Async error occured: java.lang.IllegalStateException: Cannot remove a consumer from a connection that had not been registered: ID: xxx

Re: Broker Leak

2012-12-11 Thread Gary Tully
you are looking at the need for a writeTimeoutFilter or inactivityMonitor or OS level tcp keepAlive to detect a half closed socket as the result of an abortive close of the client. On 11 December 2012 14:36, Christian Posta wrote: > I just ran your tests. > Actually, this doesn't seem to be the

Re: Broker Leak

2012-12-11 Thread Christian Posta
I just ran your tests. Actually, this doesn't seem to be the case of the consumers not going away. Looks like the connection wasn't properly cleaned up on the broker's side after the kill -9. Let me try with the trunk and see if there's anything different before I dig deeper to see why it wasn't cl

Re: Broker Leak

2012-12-10 Thread Jerry Cwiklik
Thanks, so if the subs dont go away the broker thinks it can deliver the messages to them even though the destination is gone? Seems that a temp queue removal should result/trigger a cleanup of messages associated with that destination. I dont have JUnit test. Instead, I attach sources for a simp