We have an application that has a broker(ActiveMQ 5.2, durable, in an Oracle DB) and 14 consumers (On various queues/topics) in a single JVM . We also have an outside module that runs in it's own JVM and houses a consumer on another queue. The outside module needs the main application's broker.
Our application can also be deployed as a cluster with each machine having copies of both the main application and the module. The extra brokers become slaves to the master broker (first application that was started). The cluster we test this setup on has several single core(important because on dual cores both JVMs are not competing) Windows XP machines with 2gigs of ram and java 1.6. The problem is that when the consumer of the outside module on the machine with the master broker receives a message, it starts doing heavy processing that takes 99% of the cpu and the broker seems to stop processing while this is going on. After the module is done, the broker wakes up and creates a large spike of threads (seen with the jconsole). With a cluster of 8 machines, it can easily create 400 threads and give us "java.lang.OutOfMemoryError: unable to create new native thread". These threads are simple Transport threads: Name: ActiveMQ Transport: tcp:///10.0.0.5:61616 State: RUNNABLE Total blocked: 0 Total waited: 0 Stack trace: java.net.SocketInputStream.socketRead0(Native Method) java.net.SocketInputStream.read(SocketInputStream.java:129) org.apache.activemq.transport.tcp.TcpBufferedInputStream.fill(TcpBufferedInputStream.java:50) org.apache.activemq.transport.tcp.TcpBufferedInputStream.read(TcpBufferedInputStream.java:58) java.io.DataInputStream.readInt(DataInputStream.java:370) org.apache.activemq.openwire.OpenWireFormat.unmarshal(OpenWireFormat.java:269) org.apache.activemq.transport.tcp.TcpTransport.readCommand(TcpTransport.java:209) org.apache.activemq.transport.tcp.TcpTransport.doRun(TcpTransport.java:201) org.apache.activemq.transport.tcp.TcpTransport.run(TcpTransport.java:184) java.lang.Thread.run(Thread.java:619) If we don't get the memory error, shortly after the spike(1-2 seconds), most threads disappear, but the new total is higher than before suggesting that there might have been a thread leak. Running the test again several times will eventually bust the memory. I've tried enabling thread pooling in ActiveMQ with no effect. The problem does not happen if I move the module consumer inside the main application(probably because the JVM is not competing with another). I've run out of ideas on how to fix this. Playing with the XMX and XMS settings isn't an option. Any idea on what could help or why this is happening? Maybe some configuration I didn't find? I reproduced the problem using the ActiveMQ examples. It's easy to see the thread spike with only two machines in a cluster. I'll attach the modified example in this message. Extract the zip into the example folder. Read the Readme.txt file for details on how to run the test. http://www.nabble.com/file/p22213287/example.zip example.zip -- View this message in context: http://www.nabble.com/Thread-spike-leak-when-broker-JVM-fails-to-compete-with-consumer-JVM-tp22213287p22213287.html Sent from the ActiveMQ - User mailing list archive at Nabble.com.
