Hi there, after upgrading from 2.1 to 2.2.13, Cassandra eats up all available memory within one week. The following is a diagram of the left available RAM of a single node over the course of a week: https://imgur.com/a/H9BDBxC <https://imgur.com/a/H9BDBxC>
Nodes are bare metal, 12 cores with 64GB RAM each, swapping disabled and configured with row cache disabled and auto sized key cache. Jemalloc is not configured. This is the same for all nodes. There is no noticeable cpu load on these nodes, disk io is also fine, just as tpstats. Heap dumps look fine and if you check memory usage via JMX everything seems to be in order also (max heap 8GB, heap memory usage goes down to 1GB after GC) The memory usage reported by top (res) is always around 20GB for the cassandra process which also seems to high for us (heap + off heap) as the offheap memory allocated by DirectByteBuffers is only around 400MB (reported by jxray). The available system memory slowly degrades all the time without any other memory intensive processes running. We currently drain and restart our nodes, if the available memory sinks below 5GB. That means that after a week, there is an offset of around 40GB between the reported memory usage of cassandra and the real memory usage. If we stop cassandra, nearly 100% of the system memory is available again. Has anybody observed similar memory usage with 2.2? If so, how to measure cassandras real memory usage if top and other system tools are way off? Do you know if this problem has been solved with 3.11? Cheers, Matthias --------------------------------------------------------------------- To unsubscribe, e-mail: user-unsubscr...@cassandra.apache.org For additional commands, e-mail: user-h...@cassandra.apache.org
