We have been working on filtering compaction for a month or so (though we call it deleting compaction, its implementation is as a filtering compaction strategy). The feature is nearing completion, and we have used it successfully in a limited production capacity against DSE 4.8 series.
Our use case is that our records are written anywhere between a month, up to several years before they are scheduled for deletion. Tombstones are too expensive, as we have tables with hundreds of billions of rows. In addition, traditional TTLs don't work for us because our customers are permitted to change their retention policy such that already-written records should not be deleted if they increase their retention after the record was written (or vice versa). We can clean up data more cheaply and more quickly with filtered compaction than with tombstones and traditional compaction. Our implementation is a wrapper compaction strategy for another underlying strategy, so that you can have the characteristics of whichever strategy makes sense in terms of managing your SSTables, while interceding and removing records during compaction (including cleaning up secondary indexes) that otherwise would have survived into the new SSTable. We are hoping to contribute it back to the community, so if you'd be interested in helping test it out, I'd love to hear from you. On Sat, Mar 12, 2016 at 5:12 AM Marcus Eriksson <krum...@gmail.com> wrote: > We don't have anything like that, do you have a specific use case in mind? > > Could you create a JIRA ticket and we can discuss there? > > /Marcus > > On Sat, Mar 12, 2016 at 7:05 AM, Dikang Gu <dikan...@gmail.com> wrote: > >> Hello there, >> >> RocksDB has the feature called "Compaction Filter" to allow application >> to modify/delete a key-value during the background compaction. >> https://github.com/facebook/rocksdb/blob/v4.1/include/rocksdb/options.h#L201-L226 >> >> I'm wondering is there a plan/value to add this into C* as well? Or is >> there already a similar thing in C*? >> >> Thanks >> >> -- >> Dikang >> >> >
