Upgrading to JRE7 sadly did not fix the problem. It took a few days, but system is now again in that weird state which makes little sense. I guess I need to figure out a way to reproduce it.
On 8 September 2015 at 14:36, Tim Bain <[email protected]> wrote: > Not that upgrading to JRE7 would be a bad thing, but I'd say upgrading from > ActiveMQ 5.4.2 to 5.12 is probably the more important of the two. Though > as you said, it appears to still be a problem at least in 5.9, so there's > no guarantee that upgrading ActiveMQ will fix the problem. > > My suggestion about the debugging port still stands, no matter which JRE > and ActiveMQ versions you use. > On Sep 8, 2015 2:06 AM, "Michiel Hendriks" < > [email protected]> > wrote: > > > It's not a JVM GC, but executing the broker GC (which you can execute via > > JMX) which flushes buffers, etc. And it's flushing of the cursor's > prefetch > > buffer that does the trick. > > > > The main annoying thing about this problem is that I cannot reproduce it. > > And it only happens in our production environment once in a while. I fear > > it might be a race condition in the JRE6 we use. So upgrading to JRE7 > might > > solve the problem. > > > > On 3 September 2015 at 17:57, Tim Bain <[email protected]> wrote: > > > > > For me, the weirdest part about this is the fact that a full GC somehow > > > fixes is. GCs aren't supposed to do anything that would affect the > > > application (garbage should always be dead, so there should be no > effect > > to > > > anything that's still alive), so that's really strange. > > > > > > If you start ActiveMQ with a debug port enabled, you can always grab > the > > > 5.4.2 source code and attach a debugger once you get into this > situation > > > and see if stepping through the code gives any clues to how you got > there > > > (and whether the code that "should have" returned values from method > > calls > > > really is)... > > > > > > And I hope you're planning that upgrade for as soon as you can arrange > > it. > > > > > > Tim > > > > > > On Wednesday, August 19, 2015, Michiel Hendriks < > > > [email protected]> wrote: > > > > > > > We've got a really weird problem. Some times we still have messages > on > > a > > > > queue, but they are not being consumed by the active consumers. > > > > > > > > Via JMX the QueueSize is 1 > > > > doesCursorHaveMessagesBuffered return false > > > > cursorSize returns 0 > > > > doesCursorHaveSpace returns true > > > > I can browse the message > > > > > > > > Restarting consumers has no effect. Restarting ActiveMQ, or simply > > > > performing a gc on the broker via JMX will correct it. Sending a > > message > > > to > > > > the queue will push the current pending message to a consumer, but > the > > > new > > > > message is then stuck. > > > > > > > > I've created and analyzed a memory dump. The batchList > > > (OrderedPendingList) > > > > of the QueueStorePrefetch contains a map with size 1, so > > > > doesCursorHaveMessagesBuffered should have returned true rather than > > > false. > > > > > > > > We have been unable to create a reproduction case yet, it just > happens > > at > > > > some point at stays like that until a restart or explicit gc on the > > > broker. > > > > The queue has 20 consumers (prefetch=1), and a lot of messages do > need > > to > > > > be redelivered. > > > > > > > > This is with ActiveMQ 5.4.2 running on java 1.6.0_36 > > > > (rhel-1.13.8.1.el5_11-x86_64). Yes, I know it's rather old (we don't > > have > > > > the luxury yet to upgrade). Looking at the code of a newer ActiveMQ I > > do > > > > not see differences in the path from doesCursorHaveMessagesBuffered > to > > > > OrderedPendingList.isEmpty > > > > > > > > Has anybody seen anything like this before? > > > > > > > > I did see a message from Donnell Alwyn on 04 Nov 2014 which looks > quite > > > > similar to this problem, but for AMQ 5.9. But there was no answer to > > his > > > > question. > > > > > > > > -- > > > > "If you dig a hole and it’s in the wrong place digging it deeper is > not > > > > going to help." – Seymour Chwast > > > > > > > > Michiel Hendriks, MSc > > > > MP Objects - Supply Chain Systems > > > > > > > > ✉️ [email protected] <javascript:;> > > > > 🏠 www.mp-objects.com > > > > ☎ +31 102900304 > > > > > > > > > > > > > > > -- > > "If you dig a hole and it’s in the wrong place digging it deeper is not > > going to help." – Seymour Chwast > > > > Michiel Hendriks, MSc > > MP Objects - Supply Chain Systems > > > > ✉️ [email protected] > > 🏠 www.mp-objects.com > > ☎ +31 102900304 > > > -- "If you dig a hole and it’s in the wrong place digging it deeper is not going to help." – Seymour Chwast Michiel Hendriks, MSc MP Objects - Supply Chain Systems ✉️ [email protected] 🏠 www.mp-objects.com ☎ +31 102900304
