Re: User Memory Issues

2015-08-25 Thread Tim Bain
Fair enough, that was my first thought, but I also know that sometimes you have to figure out a band-aid till the upgrade comes through. But if you're good just doing the upgrade, that sounds great to me! On Aug 24, 2015 10:50 PM, "Daniel Israel" wrote: > Hi Tim, > > I appreciate your efforts, b

Re: User Memory Issues

2015-08-24 Thread Daniel Israel
Hi Tim, I appreciate your efforts, but it really doesn't make sense for me (and you) to try to debug the issue on an environment that's 7+ years old. It's really on me to get the production system updated, I'm working on development environment now, which is where my questions come from. Don'

Re: User Memory Issues

2015-08-24 Thread Tim Bain
Yes, as long as that's on the subscription, not the topic itself. Caveat: I haven't actually looked at JMX in 5.3.2, so I'm extrapolating from what I've seen in later versions, but I think that should be right. On Aug 24, 2015 1:29 PM, "Daniel Israel" wrote: > > > > > > On 8/21/15, 5:28 AM, "tba

Re: User Memory Issues

2015-08-24 Thread jamesTheCruncher
Hi Daniel, I guess you are getting the exception in your producer and you are sending messages asynchronously. Is this right? It would be helpful if you share your activemq.xml configuration here. -- View this message in context: http://activemq.2283324.n4.nabble.com/User-Memory-Issues-tp47011

Re: User Memory Issues

2015-08-24 Thread Daniel Israel
On 8/21/15, 5:28 AM, "tbai...@gmail.com on behalf of Tim Bain" wrote: >You can tell if a consumer is slower than the rest via JMX. Look at each >subscription for the topic and see if any of them has lots of pending >messages while the rest do not. Is that the PendingQueueSize on the JMX

Re: User Memory Issues

2015-08-21 Thread Tim Bain
You can tell if a consumer is slower than the rest via JMX. Look at each subscription for the topic and see if any of them has lots of pending messages while the rest do not. If you discover that to be the case, configure a slow consumer abortion strategy to kick them off and throw away their pen

Re: User Memory Issues

2015-08-20 Thread Daniel Israel
On 8/20/15, 9:56 PM, "tbai...@gmail.com on behalf of Tim Bain" wrote: >The broker can't discard the message till all subscribers have consumed it, This is good information, thanks! I am wondering if there is a way to detect this? >so a single slow consumer will result in increased memor

Re: User Memory Issues

2015-08-20 Thread Tim Bain
The broker can't discard the message till all subscribers have consumed it, so a single slow consumer will result in increased memory/store usage and eventually PFC. You could set up a slow consumer abortion strategy if you're worried about that possibility. On Aug 20, 2015 10:13 AM, "Daniel Israe

Re: User Memory Issues

2015-08-20 Thread Daniel Israel
Oooh, that's a good idea to add another subber. I may be able to try that. Also, could something on the client side cause a backup? Or will the broker just fall through if a client hangs? On 8/20/15, 7:18 AM, "tbai...@gmail.com on behalf of Tim Bain" wrote: >Since this is happening in p

Re: User Memory Issues

2015-08-20 Thread Tim Bain
Since this is happening in production, I assume it's not acceptable to attach a debugger and set a breakpoint on the line where the exception is thrown. If you could, that would let you step into the MemoryUsage.isFull() call and see what's going on, but it'll hang the broker when it happens and I

Re: User Memory Issues

2015-08-19 Thread Daniel Israel
Forgot to attach. This is log message: 2013-03-30 22:34:42,824 [.96.47.33:34886] WARN Service - Async error occurred: javax.jms.ResourceAllocationException: Usage Manager memory limit reached javax.jms.ResourceAllocationException: Usage Manager memory limit reached

Re: User Memory Issues

2015-08-19 Thread Daniel Israel
Hi Tim, Yes, you're correct, the version I have in production is older than that. I have confirmed that it is pre-5.3.0 (It's actually a FUSE release) and the changelog in the ActiveMQ Subversion log shows the log details coming a few months later. On 8/19/15, 6:53 AM, "tbai...@gmail.com

Re: User Memory Issues

2015-08-19 Thread Tim Bain
Hmm, the error messages I'm used to seeing (from 5.8.0 and 5.10.0) look like the one in this page ( http://blogs.sourceallies.com/2014/10/activemq-memory-tuning/), which give lots of information about what limit is being hit. I guess that detailed info must have been added in a version after the o

Re: User Memory Issues

2015-08-18 Thread Daniel Israel
Hi Tim, thanks for response. Flow control is enabled, and it's configured to fail if out of memory. As noted below, the log lines in this version don't tell us which limit we're exceeding, so we're running half blind :(. Knowing average topic size would be helpful, but having individual to

Re: User Memory Issues

2015-08-18 Thread Tim Bain
Later versions give a few addition stats (such as average message size) via JMX, but that won't help you till that upgrade in production is complete. Do you have producer flow control enabled? The error you're getting doesn't match what I remember it being the last time I hit it, so I'm assuming