I'm running Cassandra 3.11 on ZFS on Linux. I use the ZFS compression
and have disabled the Cassandra SSTable compression.
I can't comment on the possible pros you've mentioned, because I didn't
do enough tests on them. I know the compression ratio is marginally
better with the same compression chunk size, but that's probably because
the files containing the metadata are also compressed.
However, it's not trouble free. As Elliott said, this is a less tested
code path. One problem I have ran into is the disk space check.
Cassandra checks for available disk space before starting a compaction,
and isn't aware of the ZFS compression. As the result of that, Cassandra
often thinks there isn't enough free disk space left and refuse to
compact the large SSTables.
My current workaround for the above issue is expensive. A large
proportion of disk space is reserved (wasted) to avoid Cassandra running
into this issue.
If you don't keep enough reserved disk space, you will eventually find
one (or more) of your Cassandra nodes ends up in constant compaction
loop, like this:
WARN [CompactionExecutor:7] 2021-02-04 19:50:33,636
CompactionTask.java:94 - insufficient space to compact all requested
files. 371463.6MB required,
BigTableReader(path='/var/lib/cassandra/data/ks/cf/md-16048-big-Data.db'),
BigTableReader(path='/var/lib/cassandra/data/ks/cf/md-16049-big-Data.db'),
BigTableReader(path='/var/lib/cassandra/data/ks/cf/md-8462-big-Data.db'),
BigTableReader(path='/var/lib/cassandra/data/ks/cf/md-8463-big-Data.db')
WARN [CompactionExecutor:7] 2021-02-04 19:50:33,636
CompactionTask.java:356 - Not enough space for compaction,
371463.6MB estimated. Reducing scope.
WARN [CompactionExecutor:7] 2021-02-04 19:50:33,636
CompactionTask.java:94 - insufficient space to compact all requested
files. 254743.16MB required,
BigTableReader(path='/var/lib/cassandra/data/ks/cf/md-16048-big-Data.db'),
BigTableReader(path='/var/lib/cassandra/data/ks/cf/md-8462-big-Data.db'),
BigTableReader(path='/var/lib/cassandra/data/ks/cf/md-8463-big-Data.db')
WARN [CompactionExecutor:7] 2021-02-04 19:50:33,636
CompactionTask.java:356 - Not enough space for compaction,
254743.16MB estimated. Reducing scope.
WARN [CompactionExecutor:7] 2021-02-04 19:50:33,637
CompactionTask.java:94 - insufficient space to compact all requested
files. 158297.66MB required,
BigTableReader(path='/var/lib/cassandra/data/ks/cf/md-16048-big-Data.db'),
BigTableReader(path='/var/lib/cassandra/data/ks/cf/md-8462-big-Data.db')
WARN [CompactionExecutor:7] 2021-02-04 19:50:33,637
CompactionTask.java:356 - Not enough space for compaction,
158297.66MB estimated. Reducing scope.
WARN [CompactionExecutor:7] 2021-02-04 20:09:05,607
CompactionTask.java:94 - insufficient space to compact all requested
files. 371463.6MB required,
BigTableReader(path='/var/lib/cassandra/data/ks/cf/md-16049-big-Data.db'),
BigTableReader(path='/var/lib/cassandra/data/ks/cf/md-16050-big-Data.db'),
BigTableReader(path='/var/lib/cassandra/data/ks/cf/md-8462-big-Data.db'),
BigTableReader(path='/var/lib/cassandra/data/ks/cf/md-8463-big-Data.db')
WARN [CompactionExecutor:7] 2021-02-04 20:09:05,608
CompactionTask.java:356 - Not enough space for compaction,
371463.6MB estimated. Reducing scope.
WARN [CompactionExecutor:7] 2021-02-04 20:09:05,608
CompactionTask.java:94 - insufficient space to compact all requested
files. 254743.16MB required,
BigTableReader(path='/var/lib/cassandra/data/ks/cf/md-16050-big-Data.db'),
BigTableReader(path='/var/lib/cassandra/data/ks/cf/md-8462-big-Data.db'),
BigTableReader(path='/var/lib/cassandra/data/ks/cf/md-8463-big-Data.db')
WARN [CompactionExecutor:7] 2021-02-04 20:09:05,608
CompactionTask.java:356 - Not enough space for compaction,
254743.16MB estimated. Reducing scope.
WARN [CompactionExecutor:7] 2021-02-04 20:09:05,608
CompactionTask.java:94 - insufficient space to compact all requested
files. 158297.66MB required,
BigTableReader(path='/var/lib/cassandra/data/ks/cf/md-16050-big-Data.db'),
BigTableReader(path='/var/lib/cassandra/data/ks/cf/md-8462-big-Data.db')
WARN [CompactionExecutor:7] 2021-02-04 20:09:05,608
CompactionTask.java:356 - Not enough space for compaction,
158297.66MB estimated. Reducing scope.
WARN [CompactionExecutor:7] 2021-02-04 20:27:28,482
CompactionTask.java:94 - insufficient space to compact all requested
files. 371463.6MB required,
BigTableReader(path='/var/lib/cassandra/data/ks/cf/md-16049-big-Data.db'),
BigTableReader(path='/var/lib/cassandra/data/ks/cf/md-16052-big-Data.db'),
BigTableReader(path='/var/lib/cassandra/data/ks/cf/md-8462-big-Data.db'),
BigTableReader(path='/var/lib/cassandra/data/ks/cf/md-8463-big-Data.db')
WARN [CompactionExecutor:7] 2021-02-04 20:27:28,482
CompactionTask.java:356 - Not enough space for compaction,
371463.6MB estimated. Reducing scope.
WARN [CompactionExecutor:7] 2021-02-04 20:27:28,482
CompactionTask.java:94 - insufficient space to compact all requested
files. 254743.16MB required,
BigTableReader(path='/var/lib/cassandra/data/ks/cf/md-16052-big-Data.db'),
BigTableReader(path='/var/lib/cassandra/data/ks/cf/md-8462-big-Data.db'),
BigTableReader(path='/var/lib/cassandra/data/ks/cf/md-8463-big-Data.db')
WARN [CompactionExecutor:7] 2021-02-04 20:27:28,482
CompactionTask.java:356 - Not enough space for compaction,
254743.16MB estimated. Reducing scope.
WARN [CompactionExecutor:7] 2021-02-04 20:27:28,482
CompactionTask.java:94 - insufficient space to compact all requested
files. 158297.66MB required,
BigTableReader(path='/var/lib/cassandra/data/ks/cf/md-16052-big-Data.db'),
BigTableReader(path='/var/lib/cassandra/data/ks/cf/md-8462-big-Data.db')
WARN [CompactionExecutor:7] 2021-02-04 20:27:28,482
CompactionTask.java:356 - Not enough space for compaction,
158297.66MB estimated. Reducing scope.
WARN [CompactionExecutor:7] 2021-02-04 20:45:56,505
CompactionTask.java:94 - insufficient space to compact all requested
files. 371463.6MB required,
BigTableReader(path='/var/lib/cassandra/data/ks/cf/md-16049-big-Data.db'),
BigTableReader(path='/var/lib/cassandra/data/ks/cf/md-16053-big-Data.db'),
BigTableReader(path='/var/lib/cassandra/data/ks/cf/md-8462-big-Data.db'),
BigTableReader(path='/var/lib/cassandra/data/ks/cf/md-8463-big-Data.db')
WARN [CompactionExecutor:7] 2021-02-04 20:45:56,505
CompactionTask.java:356 - Not enough space for compaction,
371463.6MB estimated. Reducing scope.
WARN [CompactionExecutor:7] 2021-02-04 20:45:56,505
CompactionTask.java:94 - insufficient space to compact all requested
files. 254743.16MB required,
BigTableReader(path='/var/lib/cassandra/data/ks/cf/md-16053-big-Data.db'),
BigTableReader(path='/var/lib/cassandra/data/ks/cf/md-8462-big-Data.db'),
BigTableReader(path='/var/lib/cassandra/data/ks/cf/md-8463-big-Data.db')
WARN [CompactionExecutor:7] 2021-02-04 20:45:56,505
CompactionTask.java:356 - Not enough space for compaction,
254743.16MB estimated. Reducing scope.
WARN [CompactionExecutor:7] 2021-02-04 20:45:56,505
CompactionTask.java:94 - insufficient space to compact all requested
files. 158297.66MB required,
BigTableReader(path='/var/lib/cassandra/data/ks/cf/md-16053-big-Data.db'),
BigTableReader(path='/var/lib/cassandra/data/ks/cf/md-8462-big-Data.db')
WARN [CompactionExecutor:7] 2021-02-04 20:45:56,505
CompactionTask.java:356 - Not enough space for compaction,
158297.66MB estimated. Reducing scope.
and keep going on infinitely, until someone notices it (hopefully
because of an alert, not an outage) and gets it fixed.
On 25/01/2021 15:06, Lapo Luchini wrote:
Hi,
I'm using a fairly standard install of Cassandra 3.11 on FreeBSD
12, by default filesystem is compressed using LZ4 and Cassandra tables
are compressed using LZ4 as well.
I was wondering if anybody had data about this already (or else, I
will probably do some tests myself, eventually): would it be a nice
idea to disable Cassandra compression and rely only on ZFS one?
In principle I can see some pros:
1. it's done in kernel, might be slightly faster
2. can (probably) compress more data, as I see a 1.02 compression factor
on filesystem even if I have compressed data in tables already
3. in upcoming ZFS version I will be able to use Zstd compression
(probably before Cassandra 4.0 is gold)
4. (can inspect compression directly at filesystem level)
But on the other hand application-level compression could have its
advantages.
cheers,
