Hi vignesh,
Correlation does not imply causation. I wouldn't work on the assumption
that the memory usage spikes are caused by compactions to start with.
It's best to prove the causal effect first. There's multiple ways to do
this, I'm just throwing in some ideas:
1. taking a heap dump while it is happening and analyse the heap dump
2. triggering manual compaction and observe it, repeat few times at
different time of day/day of week
3. while it is happening, stop the compaction and see whether the
memory usage immediately drop back to baseline
My preferred way is heap dump, because it will tell you exactly what
caused the memory usage spike, not just confirm or rule out one guess at
a time.
I hope this helps.
Cheers,
Bowen
On 26/02/2025 12:27, vignesh s wrote:
*Setup:*
I have a Cassandra cluster running in 3 datacenters with 3 nodes each
(total 9 nodes), hosted on GCP.
• *Replication Factor:* 3-3-3
• *Compaction Strategy:* LeveledCompactionStrategy
• *Heap Memory:* 10 GB (Total allocated memory: 32 GB)
• *Off-heap Memory:* around 4 GB
• *Workload:* ~1.5K writes/s per node, ~100 reads/s per node (both
using LOCAL_QUORUM)
*Issue:*
I am observing short-lived memory spikes where total memory usage
jumps from 44% to 85%. These occur periodically and last for a short
period.
After monitoring tpstats, I noticed that compaction threads are
running during these spikes.
While I understand that compaction is a fundamental process, these
memory spikes make capacity planning difficult.
I tried adjusting the following settings, but they did not have any
effect on the spikes:
• compaction_throughput_mb_per_sec
• concurrent_compactors
*Questions:*
1. Are there other settings I can tune to reduce memory spikes?
2. Could something else be causing these spikes apart from compaction?
Would appreciate any insights on how to smooth out memory usage.
- vignesh
