Have you cleaned up your snapshotsŠthose take extra space and don't just go away unless you delete them.
Dean On 3/28/13 11:46 AM, "Ben Chobot" <be...@instructure.com> wrote: >Are you also running 1.1.5? I'm wondering (ok hoping) that this might be >fixed if I upgrade. > >On Mar 28, 2013, at 8:53 AM, Lanny Ripple wrote: > >> We occasionally (twice now on a 40 node cluster over the last 6-8 >>months) see this. My best guess is that Cassandra can fail to mark an >>SSTable for cleanup somehow. Forced GC's or reboots don't clear them >>out. We disable thrift and gossip; drain; snapshot; shutdown; clear >>data/Keyspace/Table/*.db and restore (hard-linking back into place to >>avoid data transfer) from the just created snapshot; restart. >> >> >> On Mar 28, 2013, at 10:12 AM, Ben Chobot <be...@instructure.com> wrote: >> >>> Some of my cassandra nodes in my 1.1.5 cluster show a large >>>discrepancy between what cassandra says the SSTables should sum up to, >>>and what df and du claim exist. During repairs, this is almost always >>>pretty bad, but post-repair compactions tend to bring those numbers to >>>within a few percent of each other... usually. Sometimes they remain >>>much further apart after compactions have finished - for instance, I'm >>>looking at one node now that claims to have 205GB of SSTables, but >>>actually has 450GB of files living in that CF's data directory. No >>>pending compactions, and the most recent compaction for this CF >>>finished just a few hours ago. >>> >>> nodetool cleanup has no effect. >>> >>> What could be causing these extra bytes, and how to get them to go >>>away? I'm ok with a few extra GB of unexplained data, but an extra >>>245GB (more than all the data this node is supposed to have!) is a >>>little extreme. >> >
