On Mon, Jul 7, 2014 at 9:52 AM, Redmumba <redmu...@gmail.com> wrote: > I am having an issue on multiple machines where it's simply filling up the > disk space during what I can only assume is a compaction. For example, the > average node cluster-wide is around 900GB according to DSE > OpsCenter--however, after coming in after the three day weekend, I noticed > that there were 13 hosts that were down in the cluster. > > When I investigated, I see several huge sstables that appeared to be in > the middle of compaction when the host ran out of usable disk space: > >> >> *643G auditing-audit-tmp-jb-31207-Data.db32G >> auditing-audit-tmp-jb-32030-Data.db* >> > > From what I can tell, it is reaching the point where it is trying to > compact the ~190G ones into a combined bigger one, and is failing because > of the lack of disk space. How do I work around this? >
You mostly don't. Having enough headroom to compact is a part of Cassandra capacity planning. Are you doing a large amount of UPDATE or overwrite load which would lead to very-efficient compaction [1]? If not, your problem isn't compaction, it's that you have too much data per node. The limited workaround available here is to : 1) stop automatic minor compactions (via the thresholds) 2) run userdefinedcompaction via JMX, combining only enough of the large files to successfully compact But 2) is not likely to be very useful unless your compactions are actually reclaiming space. Probably "just" increase the size of your cluster. =Rob [1] grep % /path/to/system.log # are these percentages below 95%? if not, compaction will not reclaim disk space when done!
