I’ve seen the same thing: https://issues.apache.org/jira/browse/CASSANDRA-9577 <https://issues.apache.org/jira/browse/CASSANDRA-9577>
I’ve had cases where a restart clears the old tables, and I’ve had cases where a restart considers the old tables to be live. > On Jul 6, 2015, at 1:51 PM, Robert Coli <rc...@eventbrite.com> wrote: > > On Mon, Jul 6, 2015 at 1:46 PM, Jeff Williams <je...@wherethebitsroam.com > <mailto:je...@wherethebitsroam.com>> wrote: > 1) Cassandra version is 2.0.12. > > 2) Interesting. Looking at JMX org.apache.cassandra.db -> ColumnFamilies -> > trackcontent -> track_content -> Attributes, I get: > > LiveDiskSpaceUsed: 17788740448 <tel:17788740448>, i.e. ~17GB > LiveSSTableCount: 3 > TotalDiskSpaceUsed: 55714084629, i.e. ~55GB > > So it obviously knows about the extra disk space even though the "live" space > looks correct. I couldn't find anything to identify the actual files though. > > That's what I would expect. > > 3) So that was even more interesting. After restarting the cassandra daemon, > the sstables were not deleted and now the same JMX attributes are: > > LiveDiskSpaceUsed: 55681040579, i.e. ~55GB > LiveSSTableCount: 8 > TotalDiskSpaceUsed: 55681040579, i.e. ~55GB > > So some of my non-live tables are back live again, and obviously some of the > big ones!! > > This is permanently fatal to consistency; sorry if I was not clear enough > that if they were not live, there was some risk of Cassandra considering them > live again upon restart. > > If I were you, I would either stop the node and remove the files you know > shouldn't be live or do a major compaction ASAP. > > The behavior you just encountered sounds like a bug, and it is a rather > serious one. SSTables which should be dead being marked live is very bad for > consistency. > > Do you see any exceptions in your logs or anything? If you can repro, you > should file a JIRA ticket with the apache project... > > =Rob >
