Hi, I have a cluster of 5 nodes. We use cassandra 2.1.3. The 5 nodes use about 50-57% of the 1T SSD. One node managed to compact all its data. During one compaction this node used almost 100% of the drive. The other nodes refuse to continue compaction claiming that there is not enough disk space.
From the documentation LeveledCompactionStrategy should be able to compact my data, well at least this is what I understand. <<Size-tiered compaction requires at least as much free disk space for compaction as the size of the largest column family. Leveled compaction needs much less space for compaction, only 10 * sstable_size_in_mb. However, even if you’re using leveled compaction, you should leave much more free disk space available than this to accommodate streaming, repair, and snapshots, which can easily use 10GB or more of disk space. Furthermore, disk performance tends to decline after 80 to 90% of the disk space is used, so don’t push the boundaries.>> This is the disk usage. Node 4 is the only one that could compact everything. node0: /dev/disk1 931Gi 534Gi 396Gi 57% / node1: /dev/disk1 931Gi 513Gi 417Gi 55% / node2: /dev/disk1 931Gi 526Gi 404Gi 57% / node3: /dev/disk1 931Gi 507Gi 424Gi 54% / node4: /dev/disk1 931Gi 475Gi 456Gi 51% / When I try to compact the other ones I get this: objc[18698]: Class JavaLaunchHelper is implemented in both /Library/Java/JavaVirtualMachines/jdk1.8.0_40.jdk/Contents/Home/bin/java and /Library/Java/JavaVirtualMachines/jdk1.8.0_40.jdk/Contents/Home/jre/lib/libinstrument.dylib. One of the two will be used. Which one is undefined. error: Not enough space for compaction, estimated sstables = 2894, expected write size = 485616651726 -- StackTrace -- java.lang.RuntimeException: Not enough space for compaction, estimated sstables = 2894, expected write size = 485616651726 at org.apache.cassandra.db.compaction.CompactionTask.checkAvailableDiskSpace(CompactionTask.java:293) at org.apache.cassandra.db.compaction.CompactionTask.runMayThrow(CompactionTask.java:127) at org.apache.cassandra.utils.WrappedRunnable.run(WrappedRunnable.java:28) at org.apache.cassandra.db.compaction.CompactionTask.executeInternal(CompactionTask.java:76) at org.apache.cassandra.db.compaction.AbstractCompactionTask.execute(AbstractCompactionTask.java:59) at org.apache.cassandra.db.compaction.CompactionManager$7.runMayThrow(CompactionManager.java:512) at org.apache.cassandra.utils.WrappedRunnable.run(WrappedRunnable.java:28) I did not set the sstable_size_in_mb I use the 160MB default. Is it normal that during compaction it needs so much diskspace? What would be the best solution to overcome this problem? Thanks for your help
