I'm thinking it's unlikely that top is lying to you. Are you sure that you're measuring free memory versus available memory? Cassandra will utilise the OS page cache heavily, which will cache files in memory but leave the memory able to be reclaimed if needed. Have you checked the output of free? If available memory is still high you're perfectly fine and everything is working as expected.
On 27 August 2018 at 21:32, Matthias Pfau <matthias.p...@tutao.de> wrote: > Hi there, > after upgrading from 2.1 to 2.2.13, Cassandra eats up all available memory > within one week. The following is a diagram of the left available RAM of a > single node over the course of a week: > https://imgur.com/a/H9BDBxC <https://imgur.com/a/H9BDBxC> > > Nodes are bare metal, 12 cores with 64GB RAM each, swapping disabled and > configured with row cache disabled and auto sized key cache. Jemalloc is > not configured. > > This is the same for all nodes. There is no noticeable cpu load on these > nodes, disk io is also fine, just as tpstats. Heap dumps look fine and if > you check memory usage via JMX everything seems to be in order also (max > heap 8GB, heap memory usage goes down to 1GB after GC) > > The memory usage reported by top (res) is always around 20GB for the > cassandra process which also seems to high for us (heap + off heap) as the > offheap memory allocated by DirectByteBuffers is only around 400MB > (reported by jxray). > > The available system memory slowly degrades all the time without any other > memory intensive processes running. We currently drain and restart our > nodes, if the available memory sinks below 5GB. That means that after a > week, there is an offset of around 40GB between the reported memory usage > of cassandra and the real memory usage. > > If we stop cassandra, nearly 100% of the system memory is available again. > > Has anybody observed similar memory usage with 2.2? > > If so, how to measure cassandras real memory usage if top and other system > tools are way off? > > Do you know if this problem has been solved with 3.11? > > Cheers, > Matthias > > --------------------------------------------------------------------- > To unsubscribe, e-mail: user-unsubscr...@cassandra.apache.org > For additional commands, e-mail: user-h...@cassandra.apache.org > >
