Are you also running 1.1.5? I'm wondering (ok hoping) that this might be fixed if I upgrade.
On Mar 28, 2013, at 8:53 AM, Lanny Ripple wrote: > We occasionally (twice now on a 40 node cluster over the last 6-8 months) see > this. My best guess is that Cassandra can fail to mark an SSTable for > cleanup somehow. Forced GC's or reboots don't clear them out. We disable > thrift and gossip; drain; snapshot; shutdown; clear data/Keyspace/Table/*.db > and restore (hard-linking back into place to avoid data transfer) from the > just created snapshot; restart. > > > On Mar 28, 2013, at 10:12 AM, Ben Chobot <be...@instructure.com> wrote: > >> Some of my cassandra nodes in my 1.1.5 cluster show a large discrepancy >> between what cassandra says the SSTables should sum up to, and what df and >> du claim exist. During repairs, this is almost always pretty bad, but >> post-repair compactions tend to bring those numbers to within a few percent >> of each other... usually. Sometimes they remain much further apart after >> compactions have finished - for instance, I'm looking at one node now that >> claims to have 205GB of SSTables, but actually has 450GB of files living in >> that CF's data directory. No pending compactions, and the most recent >> compaction for this CF finished just a few hours ago. >> >> nodetool cleanup has no effect. >> >> What could be causing these extra bytes, and how to get them to go away? I'm >> ok with a few extra GB of unexplained data, but an extra 245GB (more than >> all the data this node is supposed to have!) is a little extreme. >
