Hi, Considering the following example structure:
CREATE TABLE data ( metric text, value double, time timestamp, PRIMARY KEY((metric), time) ) WITH CLUSTERING ORDER BY (time DESC) The natural inserting order is metric, value, timestamp pairs, one metric/value pair per second for example. That means creating more and more cells to the same partition, which creates a large amount of overhead and reduces the compression ratio of LZ4 & Deflate (LZ4 reaches ~0.26 and Deflate ~0.10 ratios in some of the examples I've run). Now, to improve compression ratio, how could I merge the cells on the actual Cassandra node? I looked at ICompress and it provides only byte-level compression. Could I do this on the compaction phase, by extending the DateTieredCompaction for example? It has SSTableReader/Writer facilities and it seems to be able to see the rows? I'm fine with the fact that repair run might have to do some conflict resolution as the final merged rows would be quite "small" (50kB) in size. The naive approach is of course to fetch all the rows from Cassandra - merge them on the client and send back to the Cassandra, but this seems very wasteful and has its own problems. Compared to table-LZ4 I was able to reduce the required size to 1/20th (context-aware compression is sometimes just so much better) so there are real benefits to this approach, even if I would probably violate multiple design decisions. One approach is of course to write to another storage first and once the blocks are ready, write them to Cassandra. But that again seems idiotic (I know some people are using Kafka in front of Cassandra for example, but that means maintaining yet another distributed solution and defeats the benefit of Cassandra's easy management & scalability). Has anyone done something similar? Even planned? If I need to extend something in Cassandra I can accept that approach also - but as I'm not that familiar with Cassandra source code I could use some hints. - Micke
