there is problem with size tiered compaction design. It compacts
together tables of similar size.
sometimes it might happen that you will have some sstables sitting on
disk forever (Feb 23) because no other similar sized tables were created
and probably never be. because flushed sstable is about 11-16 mb.
next level about 90 MB
then 5x 90 MB gets compacted to 400 MB sstable
and 5x400 MB ~ 2 GB
problem is that 400 MB sstable is too small to be compacted against
these 3x 720 MB ones.
-rw-r--r-- 1 root wheel 165M Feb 23 17:03 resultcache-hc-13086-Data.db
-rw-r--r-- 1 root wheel 772M Feb 23 17:04 resultcache-hc-13087-Data.db
-rw-r--r-- 1 root wheel 156M Feb 23 17:06 resultcache-hc-13091-Data.db
-rw-r--r-- 1 root wheel 716M Feb 23 17:18 resultcache-hc-13096-Data.db
-rw-r--r-- 1 root wheel 734M Feb 23 17:29 resultcache-hc-13101-Data.db
-rw-r--r-- 1 root wheel 5.0G Mar 14 09:38 resultcache-hc-13923-Data.db
-rw-r--r-- 1 root wheel 1.9G Mar 16 22:41 resultcache-hc-14084-Data.db
-rw-r--r-- 1 root wheel 1.9G Mar 21 15:11 resultcache-hc-14460-Data.db
-rw-r--r-- 1 root wheel 1.9G Mar 27 05:22 resultcache-hc-14694-Data.db
-rw-r--r-- 1 root wheel 2.0G Mar 31 04:57 resultcache-hc-14851-Data.db
-rw-r--r-- 1 root wheel 112M Mar 31 06:30 resultcache-hc-14922-Data.db
-rw-r--r-- 1 root wheel 577M Apr 1 19:25 resultcache-hc-14943-Data.db
compaction strategy needs to compact sstables by timestamp too. older
tables should have increased chance to get compacted.
for example - table from today will be compacted with other table in
range (0.5-1.5) of its size, and this range will get increased with
sstable age. - 1 month old will have range for example (0.2 - 1.8).
