The most effective way I know to determine what a Java process is doing when I can't step through with a debugger is to use a CPU sampler (JVisualVM ships with the Oracle JDK and can attach either locally or from a remote machine via JMX and RMI) to capture where the time is being spent. Let it capture data for a few minutes, then take a snapshot and dig into what the various threads are spending their time on.
Don't use the profiler on an operational system! The sampler is what you want, since it can give you good insight without measurably degrading performance, whereas the profiler will grind everything to a halt on a system as heavily loaded as yours sounds. Tim On Jun 21, 2017 1:08 PM, "sravan" <sra1em...@gmail.com> wrote: > Our batch processing applications process abut 10 Billion messages a day. > From past two months we have been experiencing an issue with ActiveMQ where > ActiveMQ delivers messages very late, sometimes messages are delivered 3 > days later. Daily, in the worst case 5% (and mostly 1%) of messages are > delivered one day later. We do not have message expiration policy by the > way > that controls what should happen when a message is not delivered for a > certain period of time. We monitored our splunk based logs and do not see > any exceptions or errors that indicate any issues with consumer > connections. > We could not turn on additional logging on ActiveMQ as that will cause a > huge hit on performance. So we mostly relied on monitoring ActiveMQ > consoles > and Dynatarce. There is no server resource utilization issues on AMQs. By > the way we have 4 AMQ nodes active in the cluster. When we monitored > activeMQ consoles we saw messages stuck in network bridge sometimes. When I > say stuck, what I mean is that message draining was extremely slow and in a > period of 2 hours I noticed only handful of messages getting drained. > Whenever we restart a AMQ node any stuck messages on that node are getting > released (I think this is a known fact to all AMQ users). What is most > frustrating is, number of stuck messages we have noticed do not correlate > to > number of messages delivered lately (i.e. delivered next day). So we are > under an impression that there could be invisible stuck messages. BTW this > issue started happening ever since we applied a Linux patch > (redhat-release-5Server-5.11.0.9 / > autofs-5.0.1-0.rc2.186.el5_11) on ActiveMQ nodes. We did some research > in > the forums to check if there is any incompatibility between AMQ 5.11 and > this Linux patch but could not find anything. Does anyone here have any > ideas suggestions how we can troubleshoot this issue further? Any inputs > would be greatly appreciated. > > > > -- > View this message in context: http://activemq.2283324.n4. > nabble.com/Messages-are-stuck-in-ActiveMQ-5-11-and- > delivered-for-after-more-than-24-hours-tp4727694.html > Sent from the ActiveMQ - User mailing list archive at Nabble.com. >
