We run a fairly small production Cassandra 2.2.4 cluster with 5 nodes on Rackspace VMs, (4 cores, 4GB RAM, SSD backed) and whilst these nodes are on the small side, day to day it has kept up with our workload fine.
We currently use SizeTieredCompactionStrategy and want to move to the LeveledStrategy. I am attempting to do this one node at a time by following http://blog.alteroot.org/articles/2015-04-20/change-cassandra-compaction-strategy-on-production-cluster.html <http://blog.alteroot.org/articles/2015-04-20/change-cassandra-compaction-strategy-on-production-cluster.html> which suggests you can do this by tweaking the compaction strategy on the fly via JMX. However, when I do this, the node soon runs out of RAM and cassandra is OOMed, or I get a Out of native memory error ( http://pastebin.com/esT2wqHz <http://pastebin.com/esT2wqHz> ) - both outcomes seem to happen randomly when the compaction is changed. I’m running the default cassandra.yaml and cassandra-env settings for everything (apart from things like seed nodes and listen addresses etc) and have also tried turning on trickle_fsync and set the fsync inteval to 100MB / 50MB which seemed to slightly increase the time before the out of memory event but not much else. Is there another way to change over compaction strategies that is less likely to use up so much RAM? Current STCS compactions complete fine, as do repairs, anticompactions etc. Chris
