Oh and as a further refinement outside of our use case. If we could group/organize the sstables by the rowkey time value or inherent TTL value, the naive version would be evenly distributed buckets into the future.
But many/most data patterns like this have "busy" data in the near term. Far out scheduled stuff would be more sparse. In our case, 50% of the data is in the first 12 hours, 50% of the remaining in the next day or two, 50% of the remaining in the next week, etc etc. So we could have a "long term" general bucket to take data far in the future. But here's the thing, if we could actively process the "long term" sstable on a regular basis into two sstables: the stuff that is still "long term" and sstables for the "near term", that could solve many general cases. The "long term" bucket could even be STCS by default, and as the near term comes into play, that is considered a different "level". Of course all this relies on the ability to look at the data in the rowkey or the TTL associated with the row. On Fri, Feb 16, 2018 at 4:17 PM, Carl Mueller <carl.muel...@smartthings.com> wrote: > We have a scheduler app here at smartthings, where we track per-second > tasks to be executed. > > These are all TTL'd to be destroyed after the second the event was > registered with has passed. > > If the scheduling window was sufficiently small, say, 1 day, we could > probably use a time window compaction strategy with this. But the window is > one-two years worth of adhoc event registration per the contract. > > Thus, the intermingling of all this data TTL'ing at the different times > since they are registered at different times means the sstables are not > written with data TTLing in the same rough time period. If they were, then > compaction would be a relatively easy process since the entire sstable > would tombstone. > > We could kind of do this by doing sharded tables for the time periods and > rotating the shards for duty, and truncating them as they are recycled. > > But an elegant way would be a custom compaction strategy that would > "window" the data into clustered sstables that could be compacted with > other similarly time bucketed sstables. > > This would require visibility into the rowkey when it came time to convert > the memtable data to sstables. Is that even possible with compaction > schemes? We would provide a requirement that the time-based data would be > in the row key if it is a composite row key, making it required. > > >
