Re: Many keyspaces pattern

2015-11-24 Thread Jack Krupansky
And DateTieredCompactionStrategy can be used to efficiently remove whole sstables when the TTL expires, but this implies knowing what TTL to set in advance. I don't know if there are any tools to bulk delete older than a specific age when DateTieredCompactionStrategy is used, but it might be a nic

Re: Many keyspaces pattern

2015-11-24 Thread Saladi Naidu
I can think of following features to solve 1. If you know the time period of after how long data should be removed then use TTL feature2. Use Time Series to model the data and use inverted index to query the data by time period? Naidu Saladi On Tuesday, November 24, 2015 6:49 AM, Jack Kr

Re: Many keyspaces pattern

2015-11-24 Thread Jack Krupansky
How often is sometimes - closer to 20% of the batches or 2%? How are you querying batches, both current and older ones? As always, your queries should drive your data models. If deleting a batch is very infrequent, maybe best to not do it and simply have logic in the app to ignore deleted batche

Many keyspaces pattern

2015-11-24 Thread Jonathan Ballet
Hi, we are running an application which produces every night a batch with several hundreds of Gigabytes of data. Once a batch has been computed, it is never modified (nor updates nor deletes), we just keep producing new batches every day. Now, we are *sometimes* interested to remove a comple