Experimenting with this could make sense, especially for inter-datacenter communication, but I caution that we might not see the same gains we see with disk compression so we might want to invest in metrics e.g. pre/post frame size (or ratio) and time spent in the compression stage as part of this change so operators can measure. I believe in 4.0 the internode refactor removed all the coalescing logic and simplified it with pulling up to 64KiB of messages into one frame [1] for the small data channel and immediately sending what you have, which means that in the common case of many small messages you might not get much data for compression to work with until you start having a very high throughput (high enough that Compressor -> Netty -> Network chain can't transit it before the event loop shows back up for a dequeue). With the post-4.0 design, we may incur the higher compression cost without seeing a meaningful payload reduction. Alternatively, for cases that enqueue enough data, we might introduce a bottleneck that prevents dequeueing fast enough. I think it's reasonable to assume the benefit is highly workload-and-network-specific.
For realtime use cases like network traffic, zstd's slower compression and much slower decompression stages (and the associated CPU costs) are often not worth the ratio improvements you get without coalescing. Coalescing on the other hand introduces complexity, latency and often shifts the bottleneck from the network to the CPU - some of the reasons it was removed iirc (although Benedict can correct me if I'm misremembering). One of the reasons zstd works well in table compression is that we can flush with lz4 and then pivot to zstd on the non-latency-sensitive compaction path, and decompression is still faster than disks most of the time. With internode networking we would be forced to pay that high compression cost in real time. Streaming might be a different story and I do remember way back when Benedict and I chatted about doing something similar to zstd adapt [2] for buffered streaming transfers to try to minimize end-to-end transfer latency of large datasets, but iirc the complexity was rather high given the way Netty pipelines work and the key metric for streaming is usually end to end transfer latency and you're hopefully rarely doing it across WAN. The problem with many AWS instance shapes these days is that the network is often much faster than your CPUs or even NVMe disks, especially when the instance is allowed to burst. All that being said, if someone has just the right workload (e.g. infrequent bursts of small messages or compressible large ones) and just the right hardware shape (e.g. abundant CPU, limited/expensive network), enabling zstd level 1 internode could make sense. I'm curious what metrics we could expose to users to let them know if enabling zstd internode is helpful or not? -Joey [1] https://github.com/apache/cassandra/blob/78817fa41b32edaff85f5fe4ff6663503f540dd8/src/java/org/apache/cassandra/net/OutboundConnection.java#L783 [2] https://github.com/facebook/zstd/blob/82d322c4973d9e2968d94047a40892bc6d9a9bdf/programs/README.md?plain=1#L205 On Tue, Aug 18, 2026 at 3:56 AM Shailaja Koppu <[email protected]> wrote: > +1 to making internode compression algo configurable. > > > On Aug 18, 2026, at 8:44 AM, Dinesh Joshi <[email protected]> wrote: > > I vaguely recall discussing this a while ago. I'm in favor of this idea. > > On Mon, Aug 17, 2026 at 3:04 AM Štefan Miklošovič <[email protected]> > wrote: > >> There is a ticket for this (1) and discussion nobody answered to (2). >> >> Is there any reason why we are compressing with lz4 only? For inter-dc >> communication where dc's are in geographically distant areas etc. I >> think that compressing it with zstd instead of lz4 would be an option >> as well as it would, presumably, transfer less data. I know there are >> caveats attached to that like (likely) increased CPU when >> de/compressing with it, especially when zstd compression levels would >> be higher than default etc. but otherwise this is worthy of giving it >> a shot? This is something people would need to opt into, of course, >> but zstd seems like a good compression algo to add for this stuff, >> especially when we declare (3) that we get the best compression ratio >> (better than lz4) at the expense of slightly worse de/compression >> times (which are tunable based on compression levels). If a traffic >> goes from Europe to Asia and it costs dozens of milliseconds then it >> is basically irrelevant if we add a few more while we compress it 30% >> less than lz4, also when inter-continental / inter-cloud traffic is >> paid for based on volume or similar. >> >> I have a working prototype locally where I can configure what the >> internode will be compressed with, I just want to check with ML this >> is something we might eventually consider adding so I can measure the >> performance and similar more deeply. >> >> (1) https://issues.apache.org/jira/browse/CASSANDRA-20488 >> (2) https://lists.apache.org/thread/dzjd3kqtwb8bncsfs36msbfv8v4kgfyy >> (3) >> https://cassandra.apache.org/doc/latest/cassandra/managing/operating/compression.html >> > >
