Can you explain a bit more what you mean by memory spikes? The defaults we ship use the same settings for min and max JVM heap size, so you should see all the memory allocated to the JVM at startup. Did you change anything here? I don't recommend doing so.
If you're referring to files in the page cache, that's unavoidable (today) Compaction reads in data and creates new files, and we do that through the pread system call, which goes through the page cache. This could be eliminated if we move compaction to direct io [1], but that's not something that can be done through config. There's no way to eliminate files being loaded into the page cache right now, and trying to do so means fighting your OS and the optimizations it provides. vmstat [2] can be helpful to understand what's going on here. There's a good reference, Linux Performance Analysis in 60,000 Milliseconds [3], which shows how to use it as well as other tools. Jon [1] https://issues.apache.org/jira/browse/CASSANDRA-19987 [2] https://www.man7.org/linux/man-pages/man8/vmstat.8.html [3] https://netflixtechblog.com/linux-performance-analysis-in-60-000-milliseconds-accc10403c55 On Wed, Feb 26, 2025 at 4:30 AM vignesh s <vigneshclou...@gmail.com> wrote: > *Setup:* > I have a Cassandra cluster running in 3 datacenters with 3 nodes each > (total 9 nodes), hosted on GCP. > • *Replication Factor:* 3-3-3 > • *Compaction Strategy:* LeveledCompactionStrategy > • *Heap Memory:* 10 GB (Total allocated memory: 32 GB) > • *Off-heap Memory:* around 4 GB > • *Workload:* ~1.5K writes/s per node, ~100 reads/s per node (both using > LOCAL_QUORUM) > *Issue:* > I am observing short-lived memory spikes where total memory usage jumps > from 44% to 85%. These occur periodically and last for a short period. > After monitoring tpstats, I noticed that compaction threads are running > during these spikes. > While I understand that compaction is a fundamental process, these memory > spikes make capacity planning difficult. > I tried adjusting the following settings, but they did not have any effect > on the spikes: > • compaction_throughput_mb_per_sec > • concurrent_compactors > *Questions:* > 1. Are there other settings I can tune to reduce memory spikes? > 2. Could something else be causing these spikes apart from compaction? > > Would appreciate any insights on how to smooth out memory usage. > > - vignesh >
