I would recommend checking first for what exact metric do you have drops: free memory or available memory. There is a common delusion about free vs available memory in Linux: https://www.linuxatemyram.com/ Overwise if you really have spikes in *used* memory and these are spikes in memory used by Cassandra process itself then I would try to decompose the memory usage: as Jon mentioned the default and expected configuration for Java heap is Xmx=Xms + -XX:+AlwaysPreTouch, so the memory for Java heap is allocated in advance and not changing dynamically from OS point of view if the settings are in place. In this case there should be some other memory region growing, here there are some resources which can help to figure out which one: https://www.youtube.com/watch?v=c755fFv1Rnk https://shipilev.net/jvm/anatomy-quarks/12-native-memory-tracking/ https://blog.arkey.fr/2020/11/30/off-heap-reconnaissance/
Regards, Dmitry On Thu, 27 Feb 2025 at 09:31, vignesh s <vigneshclou...@gmail.com> wrote: > Thanks Bowen and Jon for the clarification and suggestions! I will go > through them and dig more. > > Yes, the JVM heap size is fixed and I can see it is allocated at all > times. The spikes I am referring to happen in addition to heap allocated > memory. > I had tuned heap settings to resolve GC pause issues in the past (Thanks > to Jon for the blog on GC tuning!), but the spikes existed even before > that. So page cache might be the cause here. > > > -vignesh > > On Wed, Feb 26, 2025 at 9:01 PM Jon Haddad <j...@rustyrazorblade.com> > wrote: > >> Can you explain a bit more what you mean by memory spikes? >> >> The defaults we ship use the same settings for min and max JVM heap size, >> so you should see all the memory allocated to the JVM at startup. Did you >> change anything here? I don't recommend doing so. >> >> If you're referring to files in the page cache, that's unavoidable >> (today) Compaction reads in data and creates new files, and we do that >> through the pread system call, which goes through the page cache. This >> could be eliminated if we move compaction to direct io [1], but that's not >> something that can be done through config. There's no way to eliminate >> files being loaded into the page cache right now, and trying to do so means >> fighting your OS and the optimizations it provides. vmstat [2] can be >> helpful to understand what's going on here. There's a good >> reference, Linux Performance Analysis in 60,000 Milliseconds [3], which >> shows how to use it as well as other tools. >> >> Jon >> >> [1] https://issues.apache.org/jira/browse/CASSANDRA-19987 >> [2] https://www.man7.org/linux/man-pages/man8/vmstat.8.html >> [3] >> https://netflixtechblog.com/linux-performance-analysis-in-60-000-milliseconds-accc10403c55 >> >> >> On Wed, Feb 26, 2025 at 4:30 AM vignesh s <vigneshclou...@gmail.com> >> wrote: >> >>> *Setup:* >>> I have a Cassandra cluster running in 3 datacenters with 3 nodes each >>> (total 9 nodes), hosted on GCP. >>> • *Replication Factor:* 3-3-3 >>> • *Compaction Strategy:* LeveledCompactionStrategy >>> • *Heap Memory:* 10 GB (Total allocated memory: 32 GB) >>> • *Off-heap Memory:* around 4 GB >>> • *Workload:* ~1.5K writes/s per node, ~100 reads/s per node (both >>> using LOCAL_QUORUM) >>> *Issue:* >>> I am observing short-lived memory spikes where total memory usage jumps >>> from 44% to 85%. These occur periodically and last for a short period. >>> After monitoring tpstats, I noticed that compaction threads are running >>> during these spikes. >>> While I understand that compaction is a fundamental process, these >>> memory spikes make capacity planning difficult. >>> I tried adjusting the following settings, but they did not have any >>> effect on the spikes: >>> • compaction_throughput_mb_per_sec >>> • concurrent_compactors >>> *Questions:* >>> 1. Are there other settings I can tune to reduce memory spikes? >>> 2. Could something else be causing these spikes apart from compaction? >>> >>> Would appreciate any insights on how to smooth out memory usage. >>> >>> - vignesh >>> >> -- Dmitry Konstantinov
