On Tue, Aug 18, 2026 at 12:47 PM Štefan Miklošovič <[email protected]> wrote: > > That is a very elaborative response ... I think that we basically need > to measure it :) Is there a metric which says what is the ratio of > what we want to send before the compression and what it was compressed > to before we send it? Based on that we would know the compression > ratio. Then we could compare what lz4 and zstd compressed to over > time.
As far as I know we do _not_ have this metric. If we add zstd to internode/streaming, my main suggestion is that we measure the before/after bytes and the time spent in the compression/decompression stage and expose them as metrics. I'm not sure we will achieve much ratio improvement outside specific workloads given the 4.0 refactor that sends messages immediately on enqueue. > CPU ... well, yeah. I am really curious what the penalty would be. A > comment here (1) says that its impact on latency was "negligible" so > maybe we are just stressing too much without having the actual numbers > at hand? I think (although we can clarify) Michael is talking about disk compression. I wouldn't expect much measurable latency difference there since we hide the write latency (via memtable and flush lz4 -> recompact zstd [1]) and the read latency of zstd decompression is similar to ssd speeds. Not in C*, but in a different replication system in AWS we either needed a pre-trained dictionary or at least 64KiB of context window for zstd to start getting the ratios to justify the slow compressor (so you'd need to buffer between 64 and 128KiB of data before sending, previously known in C* as coalescing). [1] https://issues.apache.org/jira/browse/CASSANDRA-15379 > > I honestly do not know, even if we measured and realized this is a bad > idea that is also valuable info to have. I agree; the option isn't bad. I'm just worried about folks expecting the same kind of gains they got from disk compression which involves a very different equation than network frame compression. -Joey > > > (1) > https://issues.apache.org/jira/browse/CASSANDRA-20488?focusedCommentId=17941933&page=com.atlassian.jira.plugin.system.issuetabpanels%3Acomment-tabpanel#comment-17941933 > > On Tue, Aug 18, 2026 at 4:42 PM Joseph Lynch <[email protected]> wrote: > > > > Experimenting with this could make sense, especially for inter-datacenter > > communication, but I caution that we might not see the same gains we see > > with disk compression so we might want to invest in metrics e.g. pre/post > > frame size (or ratio) and time spent in the compression stage as part of > > this change so operators can measure. I believe in 4.0 the internode > > refactor removed all the coalescing logic and simplified it with pulling up > > to 64KiB of messages into one frame [1] for the small data channel and > > immediately sending what you have, which means that in the common case of > > many small messages you might not get much data for compression to work > > with until you start having a very high throughput (high enough that > > Compressor -> Netty -> Network chain can't transit it before the event loop > > shows back up for a dequeue). With the post-4.0 design, we may incur the > > higher compression cost without seeing a meaningful payload reduction. > > Alternatively, for cases that enqueue enough data, we might introduce a > > bottleneck that prevents dequeueing fast enough. I think it's reasonable to > > assume the benefit is highly workload-and-network-specific. > > > > For realtime use cases like network traffic, zstd's slower compression and > > much slower decompression stages (and the associated CPU costs) are often > > not worth the ratio improvements you get without coalescing. Coalescing on > > the other hand introduces complexity, latency and often shifts the > > bottleneck from the network to the CPU - some of the reasons it was removed > > iirc (although Benedict can correct me if I'm misremembering). One of the > > reasons zstd works well in table compression is that we can flush with lz4 > > and then pivot to zstd on the non-latency-sensitive compaction path, and > > decompression is still faster than disks most of the time. With internode > > networking we would be forced to pay that high compression cost in real > > time. > > > > Streaming might be a different story and I do remember way back when > > Benedict and I chatted about doing something similar to zstd adapt [2] for > > buffered streaming transfers to try to minimize end-to-end transfer latency > > of large datasets, but iirc the complexity was rather high given the way > > Netty pipelines work and the key metric for streaming is usually end to end > > transfer latency and you're hopefully rarely doing it across WAN. The > > problem with many AWS instance shapes these days is that the network is > > often much faster than your CPUs or even NVMe disks, especially when the > > instance is allowed to burst. > > > > All that being said, if someone has just the right workload (e.g. > > infrequent bursts of small messages or compressible large ones) and just > > the right hardware shape (e.g. abundant CPU, limited/expensive network), > > enabling zstd level 1 internode could make sense. I'm curious what metrics > > we could expose to users to let them know if enabling zstd internode is > > helpful or not? > > > > -Joey > > > > [1] > > https://github.com/apache/cassandra/blob/78817fa41b32edaff85f5fe4ff6663503f540dd8/src/java/org/apache/cassandra/net/OutboundConnection.java#L783 > > [2] > > https://github.com/facebook/zstd/blob/82d322c4973d9e2968d94047a40892bc6d9a9bdf/programs/README.md?plain=1#L205 > > > > > > On Tue, Aug 18, 2026 at 3:56 AM Shailaja Koppu <[email protected]> > > wrote: > >> > >> +1 to making internode compression algo configurable. > >> > >> > >> On Aug 18, 2026, at 8:44 AM, Dinesh Joshi <[email protected]> wrote: > >> > >> I vaguely recall discussing this a while ago. I'm in favor of this idea. > >> > >> On Mon, Aug 17, 2026 at 3:04 AM Štefan Miklošovič <[email protected]> > >> wrote: > >>> > >>> There is a ticket for this (1) and discussion nobody answered to (2). > >>> > >>> Is there any reason why we are compressing with lz4 only? For inter-dc > >>> communication where dc's are in geographically distant areas etc. I > >>> think that compressing it with zstd instead of lz4 would be an option > >>> as well as it would, presumably, transfer less data. I know there are > >>> caveats attached to that like (likely) increased CPU when > >>> de/compressing with it, especially when zstd compression levels would > >>> be higher than default etc. but otherwise this is worthy of giving it > >>> a shot? This is something people would need to opt into, of course, > >>> but zstd seems like a good compression algo to add for this stuff, > >>> especially when we declare (3) that we get the best compression ratio > >>> (better than lz4) at the expense of slightly worse de/compression > >>> times (which are tunable based on compression levels). If a traffic > >>> goes from Europe to Asia and it costs dozens of milliseconds then it > >>> is basically irrelevant if we add a few more while we compress it 30% > >>> less than lz4, also when inter-continental / inter-cloud traffic is > >>> paid for based on volume or similar. > >>> > >>> I have a working prototype locally where I can configure what the > >>> internode will be compressed with, I just want to check with ML this > >>> is something we might eventually consider adding so I can measure the > >>> performance and similar more deeply. > >>> > >>> (1) https://issues.apache.org/jira/browse/CASSANDRA-20488 > >>> (2) https://lists.apache.org/thread/dzjd3kqtwb8bncsfs36msbfv8v4kgfyy > >>> (3) > >>> https://cassandra.apache.org/doc/latest/cassandra/managing/operating/compression.html > >> > >>
