Sounds like the issue I was observing last month. ActiveMQ was randomly deciding to just "stop" delivering messages to consumers - both the shipped web console and hawtio showed things absolutely as I would expect them to be.
I ended up suspending new operations, allowed existing data to be processed and shut everything down. There should have been no data but kahadb kept a few hundred megs for reasons unknown. I moved the storage folder and started things back up. No problems since. This whole thing started after (first observed two weeks after) we upgraded from 5.7.0 to 5.10.0 and only one of our fleet experienced the problem. On 20 September 2014 01:13, Kevin Burton <bur...@spinn3r.com> wrote: > though it’s entirely possible that flow control is still enabled.. maybe > the config isn’t parsed correctly. Is there a way to verify ? It isn’t in > the UI. > > On Fri, Sep 19, 2014 at 5:09 PM, Kevin Burton <bur...@spinn3r.com> wrote: > > > - I ruled out memory issues… I tuned up memory to 3G (should be plenty) > > and enabled GC logging and there are no full GCs. > > > > - I disabled producerFlowControl… > > > > <policyEntry queue=">" > > producerFlowControl="false" > > advisoryForConsumed="true" > > gcInactiveDestinations="true" > > inactiveTimoutBeforeGC="60000”> > > > > … still experiencing the same symptoms… A restart of ActiveMQ resolve the > > issue for about 5 minutes… then my workers come to a crawl :-(… > > > > > > On Fri, Sep 19, 2014 at 3:28 PM, Tim Bain <tb...@alumni.duke.edu> wrote: > > > >> Use JConsole to look at which consumers are connected (make sure at > least > >> one is!), how many messages they're currently working on > >> (MessageCountAwaitingAcknowlegement), and whether the numbers appear to > be > >> increasing. For each queue, confirm whether the EnqueueCount is > >> increasing; if it's not, the broker isn't accepting messages due to > >> Producer Flow Control, or the producer isn't attempting to produce any > for > >> other reasons. > >> > >> On Fri, Sep 19, 2014 at 3:55 PM, Kevin Burton <bur...@spinn3r.com> > wrote: > >> > >> > We’ve been having this problem for a while now but it’s becoming more > >> > apparent as we deploy in production. > >> > > >> > Essentially all messages in ActiveMQ get stuck until I restart it. > >> Today > >> > it took three restarts… then it just started working. > >> > > >> > Here’s my config: > >> > > >> > http://pastebin.com/F5L99biA > >> > > >> > We’re using LevelDB… the box seems to be idle while this is happening. > >> > > >> > There isn’t much data either. < 1GB… should all fit into memory. > >> > > >> > It’s entirely possible it’s the client … but I’m using the official > >> > activemq client. > >> > > >> > I think I might try browsing the queue from the command line next time > >> this > >> > happens. > >> > > >> > Any other pointers to help track this down? > >> > > >> > -- > >> > > >> > Founder/CEO Spinn3r.com > >> > Location: *San Francisco, CA* > >> > blog: http://burtonator.wordpress.com > >> > … or check out my Google+ profile > >> > <https://plus.google.com/102718274791889610666/posts> > >> > <http://spinn3r.com> > >> > > >> > > > > > > > > -- > > > > Founder/CEO Spinn3r.com > > Location: *San Francisco, CA* > > blog: http://burtonator.wordpress.com > > … or check out my Google+ profile > > <https://plus.google.com/102718274791889610666/posts> > > <http://spinn3r.com> > > > > > > > -- > > Founder/CEO Spinn3r.com > Location: *San Francisco, CA* > blog: http://burtonator.wordpress.com > … or check out my Google+ profile > <https://plus.google.com/102718274791889610666/posts> > <http://spinn3r.com> >
