Re: Consumers hanging on a queue although there are messages in it

2019-05-02 Thread alan protasio
I dont know if this is exactly what is happening with you but the code below shows that if the broker memory is full, the broker will be unable to page in messages (and dispatch them). https://www.paste.org/98451 If this is your case this issue somewhat mitigated by https://issues.apache.org/jira

Re: Consumers hanging on a queue although there are messages in it

2019-05-02 Thread alan protasio
Hi... Did you guys see some log lines like the following? WARN Queue - queue://STORE, subscriptions=1, memory=0%, size=200, pending=0 cursor blocked, no space available to page in messages; usage: Usage(Main:memory:queue://STORE:memory) percentUsage=0%, usage=0, limit=1

Re: Consumers hanging on a queue although there are messages in it

2019-04-29 Thread miksonx
Hi We are facing same issue as listed above where messages are Enqueued but broker is stuck Dequeued messages. Is there any update on this ? Thank you. -- Sent from: http://activemq.2283324.n4.nabble.com/ActiveMQ-User-f2341805.html

Re: Consumers hanging on a queue although there are messages in it

2019-04-01 Thread PG
Hello There, We are facing identical issue with ActiveMQ 5.15.5 in our pre-prod environment, where the client program stops reading messages from queue and we observed this happens when the broker restarts. Untill we failover the broker to another node, client does not start consuming messages ag

Re: Consumers hanging on a queue although there are messages in it

2018-04-04 Thread Stefanic
It has been a while so here's an update. The same problem has been occurring on and off for the past two months now and there is one suspect always coming back: message grouping. We have found an tried several things and here are some of the findings: *Message groups cache* ActiveMQ defaults to a

Re: Consumers hanging on a queue although there are messages in it

2018-02-07 Thread Stefanic
Hi Tim, It happened again yesterday so I made 2 CPU sampling snapshots. They don't tell me much so if you'd like I can supply them both, contact me at snicodem AT movingintelligence D com Because it is our production environment we have reverted back to the library versions of activemq and camel

Re: Consumers hanging on a queue although there are messages in it

2018-02-02 Thread Tim Bain
If you're able to analyze the snapshot, that's great. If you want me to, I can. If you're using any custom broker plugins, it's possible that your package, class, and method names will appear in the stack trace information captured by the snapshot, so if you're using custom plugins, make sure you c

Re: Consumers hanging on a queue although there are messages in it

2018-02-02 Thread Stefanic
I will try to remember doing so, are you interested in the CPU sampling snapshot? And if so over what amount of time? (I'm not very experienced with profiling/sampling, only really done heapdump and threaddump analysis) Tim Bain wrote > Would you be able to use JVisualVM to perform CPU sampling

Re: Consumers hanging on a queue although there are messages in it

2018-02-02 Thread Tim Bain
Would you be able to use JVisualVM to perform CPU sampling while this is occurring, to find out what those threads are doing? **WARNING** Do NOT do CPU *profiling*, which would slow your broker significantly; sampling is lightweight and generally held to be safe to do against an operational process

Re: Consumers hanging on a queue although there are messages in it

2018-02-01 Thread Stefanic
Hi Haris, We also have this issue on a queue with high activity, in our case we do more than 3 million messages a day (spread out over the day so never a moment with 0 activity). But this was never an issue before we updated.. See my previous post today, if we still encounter this issue again wit

Re: Consumers hanging on a queue although there are messages in it

2018-02-01 Thread Stefanic
Hi Tim, There are no selectors, so there are no messages left behind. We have seen this three times now in production and every time when the queue is empty (after about 30 minutes of refreshing) the problem goes away. Threads behavior is rather strange, when the blocking issue is on-going we see

Re: Consumers hanging on a queue although there are messages in it

2018-01-24 Thread Haris
Hi Stefanic, Unfortunately not. It appeared again, and then went away. It /looks like/ it's related to a "lot of activity" on a queue: lots of messages and/or lots of consumer (but more probably the first: lots of messages). However, the root cause we haven't been able to find. Especially since we

Re: Consumers hanging on a queue although there are messages in it

2018-01-22 Thread Tim Bain
Are any selectors in use on the consumers of the queue that's getting "blocked"? If so, are there any messages published that don't match any of the selectors? If you take a thread dump on the broker, where do you see threads waiting? Can you reproduce this on demand in a test environment with a

Re: Consumers hanging on a queue although there are messages in it

2018-01-22 Thread Stefanic
Haris wrote > Here is something more, really weird: Almost always, the following dirty > hack "unblocked" the consumers: refreshing the queue browser page, i.e. > http://localhost:8161/admin/queues.jsp Hi Haris, Did you ever find out what caused this behavior? We recently upgraded both the Activ

Re: Consumers hanging on a queue although there are messages in it

2016-02-18 Thread Haris
Here is something more, really weird: Almost always, the following dirty hack "unblocked" the consumers: refreshing the queue browser page, i.e. http://localhost:8161/admin/queues.jsp -- View this message in context: http://activemq.2283324.n4.nabble.com/Consumers-hanging-on-a-queue-although-th