Ignore the files missing those other components, that was confirmation bias :( I was sorting by date instead of by name and just assumed that something was wrong with Cassandra. Here's an example table's SSTables, sorted by level, then by repaired status: SSTable [name=lb-432055-big-Data.db, level=0, repaired=false, instant=2017-08-22]------------ Level 1 ------------SSTable [name=lb-431497-big-Data.db, level=1, repaired=false, instant=2017-08-17] SSTable [name=lb-431496-big-Data.db, level=1, repaired=false, instant=2017-08-17] SSTable [name=lb-431495-big-Data.db, level=1, repaired=false, instant=2017-08-17] SSTable [name=lb-431498-big-Data.db, level=1, repaired=false, instant=2017-08-17] SSTable [name=lb-431499-big-Data.db, level=1, repaired=false, instant=2017-08-17] SSTable [name=lb-431501-big-Data.db, level=1, repaired=false, instant=2017-08-17] SSTable [name=lb-431503-big-Data.db, level=1, repaired=false, instant=2017-08-17] SSTable [name=lb-431500-big-Data.db, level=1, repaired=false, instant=2017-08-17] SSTable [name=lb-431502-big-Data.db, level=1, repaired=false, instant=2017-08-17] SSTable [name=lb-431504-big-Data.db, level=1, repaired=false, instant=2017-08-17] SSTable [name=lb-426107-big-Data.db, level=1, repaired=true, instant=2017-07-07] SSTable [name=lb-426105-big-Data.db, level=1, repaired=true, instant=2017-07-07] SSTable [name=lb-426090-big-Data.db, level=1, repaired=true, instant=2017-07-07] SSTable [name=lb-426092-big-Data.db, level=1, repaired=true, instant=2017-07-07] SSTable [name=lb-426094-big-Data.db, level=1, repaired=true, instant=2017-07-07] SSTable [name=lb-426096-big-Data.db, level=1, repaired=true, instant=2017-07-07] SSTable [name=lb-426104-big-Data.db, level=1, repaired=true, instant=2017-07-07] SSTable [name=lb-426102-big-Data.db, level=1, repaired=true, instant=2017-07-07] SSTable [name=lb-426100-big-Data.db, level=1, repaired=true, instant=2017-07-07]------------ Level 2 ------------SSTable [name=lb-423829-big-Data.db, level=2, repaired=false, instant=2017-06-23] SSTable [name=lb-431505-big-Data.db, level=2, repaired=false, instant=2017-08-17] SSTable [name=lb-423830-big-Data.db, level=2, repaired=false, instant=2017-06-23] SSTable [name=lb-424559-big-Data.db, level=2, repaired=false, instant=2017-06-29] SSTable [name=lb-424568-big-Data.db, level=2, repaired=false, instant=2017-06-29] SSTable [name=lb-424567-big-Data.db, level=2, repaired=false, instant=2017-06-29] SSTable [name=lb-424566-big-Data.db, level=2, repaired=false, instant=2017-06-29] SSTable [name=lb-424561-big-Data.db, level=2, repaired=false, instant=2017-06-29] SSTable [name=lb-424563-big-Data.db, level=2, repaired=false, instant=2017-06-29] SSTable [name=lb-424562-big-Data.db, level=2, repaired=false, instant=2017-06-29] SSTable [name=lb-423825-big-Data.db, level=2, repaired=false, instant=2017-06-23] SSTable [name=lb-423823-big-Data.db, level=2, repaired=false, instant=2017-06-23] SSTable [name=lb-424560-big-Data.db, level=2, repaired=false, instant=2017-06-29] SSTable [name=lb-423824-big-Data.db, level=2, repaired=false, instant=2017-06-23] SSTable [name=lb-423828-big-Data.db, level=2, repaired=false, instant=2017-06-23] SSTable [name=lb-423826-big-Data.db, level=2, repaired=false, instant=2017-06-23] SSTable [name=lb-423380-big-Data.db, level=2, repaired=false, instant=2017-06-20] SSTable [name=lb-426057-big-Data.db, level=2, repaired=true, instant=2017-07-07] SSTable [name=lb-426058-big-Data.db, level=2, repaired=true, instant=2017-07-07][...~60 more from 2017-07-07...] SSTable [name=lb-425991-big-Data.db, level=2, repaired=true, instant=2017-07-07] SSTable [name=lb-426084-big-Data.db, level=2, repaired=true, instant=2017-07-07]------------ Level 3 ------------SSTable [name=lb-383142-big-Data.db, level=3, repaired=false, instant=2016-11-19] SSTable [name=lb-383143-big-Data.db, level=3, repaired=false, instant=2016-11-19][...~40 more from 2016-11-19...] SSTable [name=lb-383178-big-Data.db, level=3, repaired=false, instant=2016-11-19] SSTable [name=lb-383188-big-Data.db, level=3, repaired=false, instant=2016-11-19] SSTable [name=lb-425948-big-Data.db, level=3, repaired=false, instant=2017-07-07] SSTable [name=lb-383179-big-Data.db, level=3, repaired=false, instant=2016-11-19] SSTable [name=lb-383175-big-Data.db, level=3, repaired=false, instant=2016-11-19][...~30 more from 2016-11-19...] SSTable [name=lb-383160-big-Data.db, level=3, repaired=false, instant=2016-11-19] SSTable [name=lb-383181-big-Data.db, level=3, repaired=false, instant=2016-11-19] SSTable [name=lb-383258-big-Data.db, level=3, repaired=true, instant=2016-11-19] SSTable [name=lb-383256-big-Data.db, level=3, repaired=true, instant=2016-11-19] SSTable [name=lb-386829-big-Data.db, level=3, repaired=true, instant=2016-11-30] SSTable [name=lb-383251-big-Data.db, level=3, repaired=true, instant=2016-11-19] SSTable [name=lb-383259-big-Data.db, level=3, repaired=true, instant=2016-11-19] SSTable [name=lb-383250-big-Data.db, level=3, repaired=true, instant=2016-11-19] SSTable [name=lb-386823-big-Data.db, level=3, repaired=true, instant=2016-11-30] SSTable [name=lb-386825-big-Data.db, level=3, repaired=true, instant=2016-11-30] SSTable [name=lb-383263-big-Data.db, level=3, repaired=true, instant=2016-11-19] SSTable [name=lb-383252-big-Data.db, level=3, repaired=true, instant=2016-11-19][...~10 more from 2016-11-19...] SSTable [name=lb-383231-big-Data.db, level=3, repaired=true, instant=2016-11-19] SSTable [name=lb-383242-big-Data.db, level=3, repaired=true, instant=2016-11-19] SSTable [name=lb-383244-big-Data.db, level=3, repaired=true, instant=2016-11-19] SSTable [name=lb-386819-big-Data.db, level=3, repaired=true, instant=2016-11-30] [...19 more from 2016-11-19...] SSTable [name=lb-425947-big-Data.db, level=3, repaired=true, instant=2017-07-07] [...8 more from 2016-11-19...] SSTable [name=lb-383262-big-Data.db, level=3, repaired=true, instant=2016-11-19]
Obviously, I cannot remember what I (or team) did on those dates, but that's what I'm trying to figure out. It looks like I performed a major compaction (nodetool compact) on or around 2016-11-19 that brought all the data into level 3. Then background compactions on 2017-06-23 and 2017-07-07 brought data into level 2. Level 1 has been growing since then. Does that seem plausible? On Monday, August 21, 2017, 5:08:44 PM PDT, kurt greaves <k...@instaclustr.com> wrote: Sorry about that Sotirios, I didn't make the connection between the two threads and this one dropped off my radar. Is it possible that there's always a compaction to be run in the "repaired" state, with that many SSTables, that unrepaired compactions are essentially "starved", considering the WrappingCompactionStrategy prioritizes the "repaired" set? Not really as the code prioritises whichever pool has the most pending tasks. See hereBesides, the repaired pool will only increase if you actually run repairs, so there is only a limited set of data that can be compacted between repairs. Cassandra has been known to leave some file handles open on occasion, I assume you've restarted Cassandra to see if it clears out the files that are invalid (the ones that only have Data and Index)?
