So no JMX stats are changing; does that mean messages aren't being produced? I had assumed you were flowing messages in but they couldn't get out so they were piling up on the broker. If you send a couple more messages, what happens?
Also, you said this is a heavily-loaded broker. Can you do anything to make it less-heavily loaded temporarily to see if this happens solely under heavy load? I'm guessing this is production we're talking about so you may not be able to. Also, what happens if you connect a new consumer in on that destination? If it gets the same behavior, you could connect a client with a debugger attached and try to find an explanation there; maybe the bug is in client code, or maybe at least something in the client-broker interaction will make the problem obvious even if it's on the broker. Tim On Apr 17, 2015 1:22 AM, "Kevin Burton" <bur...@spinn3r.com> wrote: > On Thu, Apr 16, 2015 at 8:34 PM, Tim Bain <tb...@alumni.duke.edu> wrote: > > > If that was happening you'd see the DestinationViewMBean's ExpiredCount > > increasing in the JMX counters, but only if the expiration was happening > on > > the broker; as far as I could tell, there's no stat that captures when > > messages expire while they're in the client's prefetch buffer. (But > that's > > just from empirical observation, not code inspection, so I could be wrong > > about that.) > > > > > Yes. I don’t see that. It’s still set to zero. The weird thing is that > nothing in JMX really seems odd. > > The only behavior I see is that some of my queues grow somewhat large, and > then I don’t receive any messages from them. > > If I do this from an external process, I can receive messages. Just not in > my main daemon. > > The messages have been stuck there for DAYs now. I would expect at least > *some* of them to be occasionally executed. > > This is with multiple restarts of my daemon. And I can see that my code is > calling receiveNoWait, it’s just that they never get an answer. > > > > If you're not seeing DLQ buildup, then that rules out the possibility of > > messages aging off on the broker (as long as you haven't disabled DLQing > of > > expired messages via the processExpired property on > > sharedDeadLetterStrategy). I'm not sure if it rules out the possibility > of > > them aging off on the client; I'm not sure if messages get DLQ'ed in that > > situation or simply discarded silently. We don't use the DLQ at all, so > I > > don't have any empirical observations to work from, but maybe someone > else > > on the list would know that answer... But even if it's possible, it > > certainly sounds like a bit of a long shot for explaining what you're > > seeing. > > > > Yes. We have the DLQ on and find it valuable. But I’m not seeing any > message accumulating there. > > I am going to test out using NIO to see if that fixes things. I was > thinking it might be a bug with servicing the queue. It’s worth a shot at > least. > > > > While this problem is going on, what are you seeing in JMX? Which stats > > are increasing, which ones aren't, which ones are going up slower than > the > > message count, etc.? > > > > That’s just the thing, I don’t see any of the JMX stats changing. There > are consumers on the queue I see them. But nothing really stands out. > > I’m also going to try to increase the initial redelivery time on a hunch.. > They’re not DLQing but maybe they’re just expiring automatically like you > said. It’s worth a shot. > > -- > > Founder/CEO Spinn3r.com > Location: *San Francisco, CA* > blog: http://burtonator.wordpress.com > … or check out my Google+ profile > <https://plus.google.com/102718274791889610666/posts> > <http://spinn3r.com> >
