I am not sure which version of Netty is in 3.9, but maybe you are hitting https://issues.apache.org/jira/browse/CASSANDRA-13114. I hit this in Cassandra 3.0.9 which uses Netty 4.0.23. Here is the upstream netty ticket https://github.com/netty/netty/issues/3057.
On Thu, Aug 31, 2017 at 10:15 AM, Chris Lohfink <clohfin...@gmail.com> wrote: > What version of java are you running? There is a "kinda leak" in jvm > around this you may run into, can try with -Djdk.nio.maxCachedBufferSize= > 262144 if above 8u102. You can also try increasing the size allowed for > direct byte buffers. It defaults to size of heap -XX:MaxDirectMemorySize=? > G > > Some NIO channel operations use temporary DirectByteBuffers which are >> cached in thread-local caches to avoid having to allocate / free a buffer >> at every operation. >> Unfortunately, there is no bound imposed on the size of buffers added to >> the thread-local caches. So, infrequent channel operations that require a >> very large buffer can create a native memory leak. > > > >> *Ability to limit the capacity of buffers that can be held in the >> temporary buffer cache*The system property jdk.nio.maxCachedBufferSize has >> been introduced in 8u102 to limit the memory used by the "temporary buffer >> cache." The temporary buffer cache is a per-thread cache of direct memory >> used by the NIO implementation to support applications that do I/O with >> buffers backed by arrays in the java heap. The value of the property is the >> maximum capacity of a direct buffer that can be cached. If the property is >> not set, then no limit is put on the size of buffers that are cached. >> Applications with certain patterns of I/O usage may benefit from using this >> property. In particular, an application that does I/O with large >> multi-megabyte buffers at startup but does I/O with small buffers may see a >> benefit to using this property. Applications that do I/O using direct >> buffers will not see any benefit to using this system property. >> See JDK-8147468 <http://bugs.openjdk.java.net/browse/JDK-8147468> > > > Chris > > On Thu, Aug 31, 2017 at 4:59 AM, Jonathan Baynes < > jonathan.bay...@tradeweb.com> wrote: > >> I wonder if its related to this bug (below) that’s currently unresolved, >> albeit it being reproduced way back in 2.1.11 >> >> >> >> https://issues.apache.org/jira/browse/CASSANDRA-10689 >> >> >> >> >> >> *From:* qf zhou [mailto:zhouqf2...@gmail.com] >> *Sent:* 31 August 2017 10:58 >> *To:* user@cassandra.apache.org >> *Subject:* Re: Cassandra - Nodes can't restart due to >> java.lang.OutOfMemoryError: Direct buffer memory >> >> >> >> I am using Cassandra 3.9 with cqlsh 5.0.1. >> >> 在 2017年8月31日，下午5:54，Jonathan Baynes <jonathan.bay...@tradeweb.com> 写道： >> >> >> >> again >> >> >> >> ________________________________________________________________________ >> >> This e-mail may contain confidential and/or privileged information. If >> you are not the intended recipient (or have received this e-mail in error) >> please notify the sender immediately and destroy it. Any unauthorized >> copying, disclosure or distribution of the material in this e-mail is >> strictly forbidden. Tradeweb reserves the right to monitor all e-mail >> communications through its networks. If you do not wish to receive >> marketing emails about our products / services, please let us know by >> contacting us, either by email at contac...@tradeweb.com or by writing >> to us at the registered office of Tradeweb in the UK, which is: Tradeweb >> Europe Limited (company number 3912826), 1 Fore Street Avenue London EC2Y >> 9DT. To see our privacy policy, visit our website @ www.tradeweb.com. >> > > -- - John
