Looks a lot like read repair but impossible to tell for sure
-- Jeff Jirsa > On Aug 9, 2017, at 4:34 PM, Sumanth Pasupuleti > <sumanth.pasupuleti...@gmail.com> wrote: > > My final try on pushing the attachment over. > <SSTableSlicer_output.png> > > > >> On Wed, Aug 9, 2017 at 4:01 PM, Sumanth Pasupuleti >> <sumanth.pasupuleti...@gmail.com> wrote: >> Thanks for the insights Jeff! I did go through the tickets around dropping >> expired sstables that have overlaps - based on what I understand, the only >> undesirable impact of that would be possible data resurrection. >> >> I have now attached the output of sstableslicer with the mail. Will submit a >> patch for review. >> >> Thanks, >> Sumanth >> >>> On Tue, Aug 8, 2017 at 9:49 PM, Jeff Jirsa <jji...@gmail.com> wrote: >>> The most likely cause is read repairs due to consistency level repairs >>> (digest mismatch). The only way to actually eliminate read repair is to >>> read with CL:ONE, which almost nobody does (at least in time series use >>> cases, because it implies you probably write with ALL, or run repair which >>> - as you've noted - often isn't necessary in ttl-only use cases). >>> >>> I can't see the image, but more tools for understanding sstable state are >>> never a bad thing (as long as they're generally useful and maintainable). >>> >>> For what it's worth, there are tickets in flight for being more aggressive >>> at dropping overlaps, but there are companies that use tools that stop the >>> cluster, use sstablemetadata to identify sstables we knew should be fully >>> expired, and manually remove them (/bin/rm) before starting cassandra >>> again. It works reasonably well IF (and only if) you write all data with >>> TTLs, and you can identify fully expired sstables based on maximum >>> timestamps. >>> >>> >>> >>> >>> On Tue, Aug 8, 2017 at 8:51 PM, Sumanth Pasupuleti < >>> sumanth.pasupuleti...@gmail.com> wrote: >>> >>> > Hi, >>> >> >>> >> We use TWCS in a few of the column families that have TTL based >>> >> time-series data, and no explicit deletes are issued. Over the time, we >>> >> observed the disk usage has been increasing beyond the expected levels. >>> >> >>> >> Data directory in a particular node shows SSTables that are more than >>> >> 16days old, while the bucket size is configured at 12hours, TTL is at >>> >> 15days and GC grace at 1hour. >>> >> Upon using sstableexpiredblockers, we got quite a few sets of blocking >>> >> and blocked SSTables. SSTableMetadata that is shown in the output >>> >> indicates >>> >> there is an overlap in the MinTS-MaxTS period among the blocking SSTable >>> >> and the blocked SSTables, which is preventing the older SSTables from >>> >> getting dropped/deleted. >>> >> >>> >> Following are the possible root causes we considered >>> >> >>> >> 1. Hints - old data hints getting replayed from the coordinator node. >>> >> We ruled this out since hints live for no more than 1 day based on our >>> >> configuration. >>> >> 2. External compactions - no external compactions were run, that >>> >> could cause compaction of SSTables across the TWCS buckets. >>> >> 3. Read repairs - this is ruled out as well, since we never ran >>> >> external repairs, and read repair chance on the TWCS column families >>> >> has >>> >> been set to 0. >>> >> 4. Application team writing data with older timestamp (in newer >>> >> SSTables). >>> >> >>> >> >>> >> 1. We wanted to identify the specific row keys with older timestamps >>> >> in the blocking SSTable, that could be causing this issue to >>> >> occur. We >>> >> considered using SSTable2Keys/json, however, since both the tools >>> >> involve >>> >> outputting the entire content/keys of the SSTable in the order of >>> >> the keys, >>> >> they were not helpful in this case. >>> >> 2. Since we wanted to get data on a few oldest cells with >>> >> timestamps, we created a tool mostly based off of sstable2json, >>> >> called >>> >> sstableslicer, to output 'n' top/bottom cells in an SSTable, >>> >> ordered either >>> >> on writetime/localDeletionTime. This helped us identify the >>> >> specific cells >>> >> in new SSTables with older timestamps, which further helped in >>> >> debugging on >>> >> the application end. From application team perspective, however, >>> >> writing >>> >> data with old timestamp is not a possible scenario. >>> >> >>> >> 3. Below is a sample output of sstableslicer >>> > [image: Inline image 2] >>> > >>> > >>> >> Looking for suggestions, especially around following two things: >>> >> >>> >> 1. Did we miss any other case in TWCS that could be causing such >>> >> overlap? >>> >> 2. Does sstableslicer seem valuable, to be included in Apache C*? If >>> >> yes, I shall create a JIRA and submit a PR/patch for review. >>> >> >>> >> C* version we use is 2.1.17. >>> > >>> > Thanks, >>> >> Sumanth >>> >> >>> > >> >
