Hmm, too bad pulling the obvious threads didn't yield anything. If you start seeing this happen regularly, maybe you could run a few of the clients that are likely to hit the problem with the debugging port open so you can connect in after the fact and try to figure out what's going on...
And it's good to know that load by itself isn't the sole cause of the problem; it may contribute to hitting some other bug due to a race condition or timeout of some sort, but it isn't the sole cause. Tim On Fri, Apr 17, 2015 at 11:47 AM, Kevin Burton <bur...@spinn3r.com> wrote: > On Fri, Apr 17, 2015 at 5:24 AM, Tim Bain <tb...@alumni.duke.edu> wrote: > > > So no JMX stats are changing; does that mean messages aren't being > > produced? I had assumed you were flowing messages in but they couldn't > get > > out so they were piling up on the broker. If you send a couple more > > messages, what happens? > > > > > Oh. I killed the process that produces them. It would just end up > consuming all memory since we’re using the memory store. > > > > Also, you said this is a heavily-loaded broker. Can you do anything to > > make it less-heavily loaded temporarily to see if this happens solely > under > > heavy load? I'm guessing this is production we're talking about so you > may > > not be able to. > > > > > It’s a good idea and I took that path and reduced load by like , 50% .. > .and that didn’t resolve it. > > I think it might have something to do with the fact that the broker or > client is getting overloaded with so many sessions / consumers to a large > number of consumers. > > I know the broker can still serve messages from these queues because I can > read them manually myself from another client. > > > > Also, what happens if you connect a new consumer in on that destination? > > If it gets the same behavior, you could connect a client with a debugger > > attached and try to find an explanation there; maybe the bug is in client > > code, or maybe at least something in the client-broker interaction will > > make the problem obvious even if it's on the broker. > > > > That’s what I thought as well but a new client is able to read the > messages. > > -- > > Founder/CEO Spinn3r.com > Location: *San Francisco, CA* > blog: http://burtonator.wordpress.com > … or check out my Google+ profile > <https://plus.google.com/102718274791889610666/posts> > <http://spinn3r.com> >
