Yes, however, for anyone dealing with this in the future, adding a second EBS volume is much quicker during a crisis. Granted, I haven't yet gone through the process of merging the two volumes, but it sounds like it should be fairly straight forward. The tradeoff, of course, is spending the time moving data later versus crisis mode (when we had 3 nodes, all running out of space during cleanup/compaction).
On Wed, Feb 17, 2016 at 11:42 AM, Ben Bromhead <b...@instaclustr.com> wrote: > +1 to checking for snapshots. Cassandra by default will automatically > snapshot tables before destructive actions like drop or truncate. > > Some general advice regarding cleanup. Cleanup will result in a temporary > increase in both disk I/O load and disk space usage (especially with STCS). > It should only be used as part of a planned increase in capacity when you > still have plenty of disk space left on your existing nodes. > > If you are running Cassandra in the cloud (AWS, Azure etc) you can add an > EBS volume, copy your sstables to it then bind mount it to the troubled CF > directory. This will give you some emergency disk space to let compaction > and cleanup do its thing safely. > > On Tue, 16 Feb 2016 at 10:57 Robert Coli <rc...@eventbrite.com> wrote: > >> On Sat, Feb 13, 2016 at 4:30 PM, Branton Davis < >> branton.da...@spanning.com> wrote: >> >>> We use SizeTieredCompaction. The nodes were about 67% full and we were >>> planning on adding new nodes (doubling the cluster to 6) soon. >>> >> >> Be sure to add those new nodes one at a time. >> >> Have you checked for, and cleared, old snapshots? Snapshots are >> automatically taken at various times and have the unusual property of >> growing larger over time. This is because they are hard links of data files >> and do not take up disk space of their own until the files they link to are >> compacted into new files. >> >> =Rob >> >> > -- > Ben Bromhead > CTO | Instaclustr <https://www.instaclustr.com/> > +1 650 284 9692 > Managed Cassandra / Spark on AWS, Azure and Softlayer >
