> I recently ran into a problem during a repair operation where my nodes > completely ran out of space and my whole cluster was... well, clusterfucked. > > I want to make sure how to prevent this problem in the future.
Depending on which version you're on, you may be seeing this: https://issues.apache.org/jira/browse/CASSANDRA-1674 But regardless, disk space variations is a fact of life with cassandra. Off the top of my head I'm not ready to say what the expectations are with respect to repair under all circumstances. Anyone? > Should I make sure that at all times every node is under 50% of its disk > space? Are there any normal day-to-day operations that would cause the any > one node to double in size that I should be aware of? If on or more nodes to > surpass the 50% mark, what should I plan to do? Major compactions can potentially double the amount of disk, if you have a single large column family that contributes almost all disk space. For such clusters, regular background compaction can indeed cause a doubling when the compaction happened to be a major one (i.e., happened to include all sstables). -- / Peter Schuller
