Hi, I have a Cassandra cluster(3.11.4) that does heavy writing work. (14k~16k write throughput per second per node)
Nodes are physical machine in data center. Number of nodes are 30. Each node has three data disks mounted. A few days ago, a QueryTimeout problem occurred due to Full GC. So, referring to this blog(https://thelastpickle.com/blog/2018/04/11/gc-tuning.html <https://thelastpickle.com/blog/2018/04/11/gc-tuning.html>), it seemed to have been solved by changing the memtable_allocation_type to offheap_objects. But today, I got an alarm saying that some nodes are using more than 90% of physical memory. (115GiB /125GiB) Native memory usage of some nodes is gradually increasing. All tables use TWCS, and TTL is 2 weeks. Below is the applied jvm option. -Xms31g -Xmx31g -XX:+UseG1GC -XX:G1RSetUpdatingPauseTimePercent=5 -XX:MaxGCPauseMillis=500 -XX:InitiatingHeapOccupancyPercent=70 -XX:ParallelGCThreads=24 -XX:ConcGCThreads=24 … What additional things can I try? I am looking forward to the advice of experts. Regards.
