How much free space do you have, and how big is the table? Switching to LCS is another option.
-- Jeff Jirsa > On Sep 10, 2018, at 12:09 PM, Oleksandr Shulgin > <oleksandr.shul...@zalando.de> wrote: > >> On Mon, 10 Sep 2018, 19:40 Jeff Jirsa, <jji...@gmail.com> wrote: >> I think it's important to describe exactly what's going on for people who >> just read the list but who don't have context. This blog does a really good >> job: >> http://thelastpickle.com/blog/2016/07/27/about-deletes-and-tombstones.html , >> but briefly: >> >> - When a TTL expires, we treat it as a tombstone, because it may have been >> written ON TOP of another piece of live data, so we need to get that >> deletion marker to all hosts, just like a manual explicit delete >> - Tombstones in sstable A may shadow data in sstable B, so doing anything on >> just one sstable MAY NOT remove the tombstone - we can't get rid of the >> tombstone if sstable A overlaps another sstable with the same partition >> (which we identify via bloom filter) that has any data with a lower >> timestamp (we don't check the sstable for a shadowed value, we just look at >> the minimum live timestamp of the table) >> >> "nodetool garbagecollect" looks for sstables that overlap (partition keys) >> and combine them together, which makes tombstones past GCGS purgable and >> should remove them (and data shadowed by them). >> >> If you're on a version without nodetool garbagecollection, you can >> approximate it using user defined compaction ( >> http://thelastpickle.com/blog/2016/10/18/user-defined-compaction.html ) - >> it's a JMX endpoint that let's you tell cassandra to compact one or more >> sstables together based on parameters you choose. This is somewhat like >> upgradesstables or scrub, but you can combine sstables as well. If you >> choose candidates intelligently (notably, oldest sstables first, or sstables >> you know overlap), you can likely manually clean things up pretty quickly. >> At one point, I had a jar that would do single sstable at a time, oldest >> sstable first, and it pretty much worked for this purpose most of the time. >> >> If you have room, a "nodetool compact" on stcs will also work, but it'll >> give you one huge sstable, which will be unfortunate long term (probably >> less of a problem if you're no longer writing to this table). > > > That's a really nice refresher, thanks Jeff! > > From the nature of the data at hand and because of the SizeTiered compaction, > I would expect that more or less all tables do overlap with each other. > > Even if we would be able to identify the overlapping ones (how?), I expect > that we would have to do an equivalent of the major compaction, but (maybe) > in multiple stages. Not sure that's really worth the trouble for us. > > Thanks, > -- > Alex > >>> On Mon, Sep 10, 2018 at 10:29 AM Charulata Sharma (charshar) >>> <chars...@cisco.com.invalid> wrote: >>> Scrub takes a very long time and does not remove the tombstones. You should >>> do garbage cleaning. It immediately removes the tombstones. >>> >>> >>> >>> Thaks, >>> >>> Charu >>> >>> >>> >>> From: Oleksandr Shulgin <oleksandr.shul...@zalando.de> >>> Reply-To: "user@cassandra.apache.org" <user@cassandra.apache.org> >>> Date: Monday, September 10, 2018 at 6:53 AM >>> To: "user@cassandra.apache.org" <user@cassandra.apache.org> >>> Subject: Drop TTLd rows: upgradesstables -a or scrub? >>> >>> >>> >>> Hello, >>> >>> >>> >>> We have some tables with significant amount of TTLd rows that have expired >>> by now (and more gc_grace_seconds have passed since the TTL). We have >>> stopped writing more data to these tables quite a while ago, so background >>> compaction isn't running. The compaction strategy is the default >>> SizeTiered one. >>> >>> >>> >>> Now we would like to get rid of all the droppable tombstones in these >>> tables. What would be the approach that puts the least stress on the >>> cluster? >>> >>> >>> >>> We've considered a few, but the most promising ones seem to be these two: >>> `nodetool scrub` or `nodetool upgradesstables -a`. We are using Cassandra >>> version 3.0. >>> >>> >>> >>> Now, this docs page recommends to use upgradesstables wherever possible: >>> https://docs.datastax.com/en/cassandra/3.0/cassandra/tools/toolsScrub.html >>> >>> What is the reason behind it? >>> >>> >>> >>> From source code I can see that Scrubber the class which is going to drop >>> the tombstones (and report the total number in the logs): >>> https://github.com/apache/cassandra/blob/cassandra-3.0/src/java/org/apache/cassandra/db/compaction/Scrubber.java#L308 >>> >>> >>> >>> I couldn't find similar handling in the upgradesstables code path. Is the >>> assumption correct that this one will not drop the tombstone as a side >>> effect of rewriting the files? >>> >>> >>> >>> Any drawbacks of using scrub for this task? >>> >>> >>> >>> Thanks, >>> -- >>> >>> Oleksandr "Alex" Shulgin | Senior Software Engineer | Team Flux | Data >>> Services | Zalando SE | Tel: +49 176 127-59-707 >>> >>>
