Hi,
We've been run activemq 5.10.2 with thousands of clients in our production
for quite a long time. Several days ago, we upgraded to the newer version
5.13.3. After running for several days, we saw the OutOfMemoryError and
activemq hung up in busy GC:
2016-06-06 02:01:54,238 | ERROR | Error in thread 'ActiveMQ
BrokerService[localhost] Task-10229' |
org.apache.activemq.thread.TaskRunnerFactory | ActiveMQ
BrokerService[localhost] Task-10229
java.lang.OutOfMemoryError: Direct buffer memory
at java.nio.Bits.reserveMemory(Bits.java:658)[:1.7.0_60]
at
java.nio.DirectByteBuffer.<init>(DirectByteBuffer.java:123)[:1.7.0_60]
at java.nio.ByteBuffer.allocateDirect(ByteBuffer.java:306)[:1.7.0_60]
at sun.nio.ch.Util.getTemporaryDirectBuffer(Util.java:174)[:1.7.0_60]
at sun.nio.ch.IOUtil.write(IOUtil.java:58)[:1.7.0_60]
at
sun.nio.ch.SocketChannelImpl.write(SocketChannelImpl.java:487)[:1.7.0_60]
at
org.apache.activemq.transport.nio.NIOOutputStream.write(NIOOutputStream.java:206)[activemq-client-5.13.3.jar:5.13.3]
at
org.apache.activemq.transport.nio.NIOOutputStream.write(NIOOutputStream.java:116)[activemq-client-5.13.3.jar:5.13.3]
at java.io.DataOutputStream.write(DataOutputStream.java:107)[:1.7.0_60]
at
org.apache.activemq.openwire.v10.BaseDataStreamMarshaller.tightMarshalByteSequence2(BaseDataStreamMarshaller.java:431)[activemq-client-5.13.3.jar:5.13.3]
at
org.apache.activemq.openwire.v10.MessageMarshaller.tightMarshal2(MessageMarshaller.java:182)[activemq-client-5.13.3.jar:5.13.3]
at
org.apache.activemq.openwire.v10.ActiveMQMessageMarshaller.tightMarshal2(ActiveMQMessageMarshaller.java:89)[activemq-client-5.13.3.jar:5.13.3]
at
org.apache.activemq.openwire.v10.ActiveMQTextMessageMarshaller.tightMarshal2(ActiveMQTextMessageMarshaller.java:89)[activemq-client-5.13.3.jar:5.13.3]
at
org.apache.activemq.openwire.OpenWireFormat.tightMarshalNestedObject2(OpenWireFormat.java:426)[activemq-client-5.13.3.jar:5.13.3]
at
org.apache.activemq.openwire.v10.BaseDataStreamMarshaller.tightMarshalNestedObject2(BaseDataStreamMarshaller.java:135)[activemq-client-5.13.3.jar:5.13.3]
at
org.apache.activemq.openwire.v10.MessageDispatchMarshaller.tightMarshal2(MessageDispatchMarshaller.java:105)[activemq-client-5.13.3.jar:5.13.3]
at
org.apache.activemq.openwire.OpenWireFormat.marshal(OpenWireFormat.java:234)[activemq-client-5.13.3.jar:5.13.3]
at
org.apache.activemq.transport.tcp.TcpTransport.oneway(TcpTransport.java:193)[activemq-client-5.13.3.jar:5.13.3]
I searched the error on web and found it's nio direct buffer limit problem
and we can increase the jvm setting -XX:MaxDirectMemorySize. The default
value of this setting is pretty small, only 64m . We never set this value
with 5.10.2. So I guessed the new version required more memory than the old
version so I set this value to 4g. I thought it should be more than enough.
We also started to monitor the direct buffer memory usage which we hadn't
done before.
Unfortunately, after one day we saw the direct buffer usage reached 4g and
activemq again was busy GC'ing. After a while we saw the same
OutOfMemoryError again. I then thought there might be some memory leak in
the new version. So I reverted back to 5.10.2, without the
MaxDirectMemorySize setting. Interesting thing happened. The old version
runs well but the actual direct buffer it uses is far more than 4g. It's
around 10g all the time!
So I have 2 questions here:
1. How much direct buffer activemq actually needs, or based what should we
set this memory setting?
2. How can activemq 5.10.2 ignores the -XX:MaxDirectMemorySize while the new
version can not?
--
View this message in context:
http://activemq.2283324.n4.nabble.com/nio-buffer-memory-problem-tp4712706.html
Sent from the ActiveMQ - User mailing list archive at Nabble.com.
