Hi all, We've been having a strange issue where the CPU load on our ActiveMQ server increases continuously, only to suddenly drop off like clockwork at noon and midnight sharp. We are using JDK 1.7_04 with the G1 collector.
The server is receiving barely any traffic from 8pm-5am, so the CPU load appears completely unrelated to the rate at which we process messages through ActiveMQ. Also, 10am-4pm is when it is processing the most messages, so it's odd that it's at the lowest CPU usage around 12:10pm in the middle of this peak time. At first I suspected run-away NIO threads that are furiously polling, but it does appear that ActiveMQ is cleaning up and constantly re-allocating new "NIO Worker" threads and in examining their stacktrace they are all happily spending most of their time here: sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:226) java.util.concurrent.SynchronousQueue$TransferStack.awaitFulfill(SynchronousQueue.java:460) java.util.concurrent.SynchronousQueue$TransferStack.transfer(SynchronousQueue.java:359) java.util.concurrent.SynchronousQueue.poll(SynchronousQueue.java:942) java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1043) java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1103) java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:603) java.lang.Thread.run(Thread.java:722) Here you can see the "Top Threads" using CPU as reported by the top threads plug-in for JConsole. The top threads are always mostly the "NIO Worker" threads. http://activemq.2283324.n4.nabble.com/file/n4654328/alertmq01_nio_cpu_12hrs.jpg Here you can see like clockwork the CPU drops at noon and midnight. The 23rd is the exception, where the drop-offs that didn't happen at noon and midnight were manual restarts of the server trying various additional settings. http://activemq.2283324.n4.nabble.com/file/n4654328/cpu-week.png For kicks, I tried the 5.7-SNAPSHOT from July 23rd. It didn't affect the gradual increase in CPU load. We are using "-Dorg.apache.activemq.UseDedicatedTaskRunner=false" Here's the line from activemq.xml where we define the transportConnector: <transportConnector name="openwire" uri="nio://0.0.0.0:61616?transport.closeAsync=false&soTimeout=60000&wireFormat.maxInactivityDurationInitalDelay=60000&keepAlive=true"/> I'm likely to try reverting back to BIO next as I have a hunch this might be similar to the NIO problems that Jetty used to have back on early JDK 1.6 flavors. I figure to try the forums to see if anyone had any thoughts... Cheers, Mike -- View this message in context: http://activemq.2283324.n4.nabble.com/ActiveMQ-5-6-CPU-steadily-increases-but-drops-at-noon-and-midnight-exactly-tp4654328.html Sent from the ActiveMQ - User mailing list archive at Nabble.com.
