For now, let's focus on the garbage collection behavior. CMS has a well-documented failure mode where fragmentation of the Old Gen space can result in the JVM being unable to allocate a new large object even though the heap isn't full. That might be what's going on here, or it might not be related at all. But the fact that you have an 8-minute (!!!) full GC seems like there might be something here.
JConsole will give you good details about the GC history and the breakdown of the heap. Can you see if it shows heavy GC activity leading up to and after the broker becomes unresponsive? Also, you may want to try switching to G1GC, since it doesn't have CMS's failure mode. Tim On May 18, 2017 2:09 AM, "Shobhana" <shobh...@quickride.in> wrote: I observed closely for last 3 days, but couldn't identify anything wrong with CPU, memory, Network I/O, Disk I/O, etc a) CPU usage on this EC2 instance (has 8 vCPUs) has never crossed 35% b) Memory usage varies between 1GB to 18 GB (Total available is 32 GB and Xmx assigned for this broker process is 26 GB) c) Thread dumps don't show any blocked threads d) Logs (enabled at INFO level) don't show any errors, except for occasional "Failed to remove inactive destination. Destination still has an active subscription". Is there any log to indicate producer flow control has kicked in? In this morning run, there was 1 full GC (we use CMS GC) before the issue popped up and another full GC just 1 second after the issue. The first full GC took about 8.17 secs. Does this indicate any trouble? I enabled JMX to check the no of messages pending. The JMX console just shows message statistics for each destination; However in my setup, there are thousands of topics created and thousands of durable subscribers. The JMX console couldn't even load all of them. So couldn't get how many messages were pending. Is there any other way to get total no of messages pending delivery? Is our usage of ActiveMQ following any of the anti-patterns? We create thousands of connections, hundreds of thousands of topics and durable subscribers to exchange MQTT messages. Is this usage not recommended? -- View this message in context: http://activemq.2283324.n4. nabble.com/ActiveMQ-broker-becomes-unresponsive-after- sometime-tp4725278p4726317.html Sent from the ActiveMQ - User mailing list archive at Nabble.com.
