Re: Several hundred messages occupies almost 30G storage

2015-03-01 Thread Tim Bain
I think I'd call that Option #2; you don't actually care about reprocessing each and every one of those messages because you need its content, you just need to make sure you fix whatever the bug was (and yes, clearly you need to test the code change, but it doesn't have to be with exactly the data

Re: Bug in TopicSubscription (wrong log message?)

2015-03-01 Thread Tim Bain
The statement is right, because the other 1x is in the consumer's prefetch buffer. It's a bit confusing that the word "pending" in the log line counts both the pending queue on the broker and the prefetch buffer on the client, but the intent is definitely right. If you have a better word ("uncons

Re: Are message actually evicted/deleted from a queue on slow consumers?

2015-03-01 Thread Tim Bain
Nope, slow consumer strategies and pending message limit strategies are orthogonal; messages are discarded because there are two many of them pending and you've configured a pending message limit strategy, not necessarily because consumers are considered slow. (In fact, depending on your settings,

Re: Simple JMX metric for the number of slow consumers?

2015-03-01 Thread Tim Bain
Ahh, now I understand the confusion, because I had to re-read that section of the web page when I read it the first time. All of that information is only in the context of the pending message limit strategies described above it; if you've specified one of them (which must be done manually; that's

Re: Documentation on the default slow connection strategy.

2015-03-01 Thread Tim Bain
There isn't a default. You have to explicitly enable one of the two available strategies (AbortSlowConsumerStrategy or AbortSlowAckConsumerStrategy); if you don't do that, there will be no attempt to identify consumers as slow nor to do anything in response to them being slow (which involves loggi

Re: Several hundred messages occupies almost 30G storage

2015-03-01 Thread Rural Hunter
Option 3 is the best fit for my use case. It's a developing system so I need someone to fix the code or another system if it fails. I need to reporocess those failed messages to verify the fix. I considered to store those messages in db earlier but the db storing process could be one of the origina

Bug in TopicSubscription (wrong log message?)

2015-03-01 Thread Kevin Burton
This looks like a but. The log message is hard wired to say “*twice* its prefetch limit pending” but the if statement doesn’t have anything like a 2x modifier. So perhaps this log line just needs to be re-written? If you guys agree I’ll create a JIRA for it. if (info.getPrefetchSize

Re: Are message actually evicted/deleted from a queue on slow consumers?

2015-03-01 Thread Kevin Burton
And I can’t tell if it’s saying that messages are evicted from the entire QUEUE or just messages target for that consumer. I’m looking at TopicSubscription to try to figure it out and it’s unclear if it’s just for messages targeted to a specific consumer which are then placed *back* into the queue

Are message actually evicted/deleted from a queue on slow consumers?

2015-03-01 Thread Kevin Burton
Message eviction seems to be not well documented so I’m confused. If I have a slow consumer, does the broker actually *delete* messages? So if I have an errant consumer, configured incorrectly, this will impact production because messages are just removed from the queue? -- Founder/CEO Spinn3

Re: Simple JMX metric for the number of slow consumers?

2015-03-01 Thread Kevin Burton
http://activemq.apache.org/slow-consumer-handling.html > We have a MessageEvictionStrategy which is used to decide which message should be evicted on a slow consumer. I’m also unclear why / if this is happening. If I have a slow consumer I definitely don’t just want messages being removed from my

Re: Simple JMX metric for the number of slow consumers?

2015-03-01 Thread Tim Bain
Are you talking about message expiration? Or do you have another definition for message eviction that I'm not thinking of? On Sun, Mar 1, 2015 at 3:24 PM, Kevin Burton wrote: > Does the same apply to evictions? > > If I have messages begin evicted by ActiveMQ I definitely want to know it! > :)

Re: Simple JMX metric for the number of slow consumers?

2015-03-01 Thread Kevin Burton
Does the same apply to evictions? If I have messages begin evicted by ActiveMQ I definitely want to know it! :) On Sun, Mar 1, 2015 at 2:07 PM, Kevin Burton wrote: > Cool! > > I think broker level at a minimum but I imagine having both on the > destination level would be good too. > > https:/

Documentation on the default slow connection strategy.

2015-03-01 Thread Kevin Burton
This doesn’t seem to be documented very well and I’m having trouble figuring out what the default slow connection strategy is for consumers. http://activemq.apache.org/per-destination-policies.html Perhaps printing some of the defaults on startup as they change between ActiveMQ versions and showi

Re: Simple JMX metric for the number of slow consumers?

2015-03-01 Thread Kevin Burton
Cool! I think broker level at a minimum but I imagine having both on the destination level would be good too. https://issues.apache.org/jira/browse/AMQ-5624 On Sun, Mar 1, 2015 at 1:59 PM, Tim Bain wrote: > This sounds like a useful feature; submit an Enhancement in JIRA for it... > > Were y

Re: Simple JMX metric for the number of slow consumers?

2015-03-01 Thread Tim Bain
This sounds like a useful feature; submit an Enhancement in JIRA for it... Were you looking for a broker-level count (count of slow consumers on all destinations) or a destination-level count? Or both? (Implementing both seems most useful to me, and not particularly hard to do.) Tim On Sun,

Simple JMX metric for the number of slow consumers?

2015-03-01 Thread Kevin Burton
According to this: http://activemq.apache.org/jmx.html There are no simple metrics for the number of slow consumers. Is there an easy way to figure this out? Would be nice for monitoring systems so that alerts can be triggered. Right now you have to enumerate each subscriber but it takes 200ms

Re: Durable subscription not receiving messages

2015-03-01 Thread Peter Hicks
Hi Tim On 28 Feb 2015, at 16:41, Tim Bain wrote: > Are messages being sent to the topic at the time the consumer reconnects, > or is it just consuming a backlog of already-sent messages? Yes, additional messages are being sent on to the topic when the consumer reconnects - around 1-2 per secon

Re: ActiveMQ refusing to send messages to my clients.

2015-03-01 Thread Tim Bain
I'd start by trying to figure out whether what's slow is the broker's delivery of messages to the ActiveMQ client code or the ActiveMQ client code's delivery of messages to your code, which is what those JMX stats would help you figure out. You say you're not seeing any log lines about slow consum

Re: ActiveMQ refusing to send messages to my clients.

2015-03-01 Thread Kevin Burton
Thanks.. I’m investigating whether this is slow consumer activity but I don’t have any log error messages marking them as slow. On Sun, Mar 1, 2015 at 6:59 AM, Tim Bain wrote: > Look in JMX to view all the consumers for a queue; for each one, look at > the dispatched count and see whether it's

Re: Several hundred messages occupies almost 30G storage

2015-03-01 Thread Tim Bain
So what do you plan to do with those failed messages? I see three possibilities, but have no idea which of them is the one (or ones) that apply to your situation. 1. Are you going to reprocess them as-is because whatever failed is assumed to just be some transient failure that won't fail th

Re: Several hundred messages occupies almost 30G storage

2015-03-01 Thread Rural Hunter
I know it is the best way that I consume the messages timely for this queue. It's just the queue I cares about the failed messages so I could processthem timely. What about other queues if I don't care much about the failed messages and they go to the real dead letter queues in the same manner? 20

Re: ActiveMQ refusing to send messages to my clients.

2015-03-01 Thread Tim Bain
Look in JMX to view all the consumers for a queue; for each one, look at the dispatched count and see whether it's around zero, around your prefetch buffer size, or somewhere in the middle. If they're all near zero, that means the broker isn't delivering messages to the clients' prefetch buffer fa