[
https://issues.apache.org/jira/browse/CASSANDRA-19979?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=18111967#comment-18111967
]
Jon Haddad edited comment on CASSANDRA-19979 at 9/6/26 1:34 AM:
----------------------------------------------------------------
I've got a WIP patch for this. Going to get some numbers from a cluster to back
it all up next. I've added quite a few tests to this to ensure there's no
regressions, but I'll still put it through the wringer on real clusters.
Note: The branch only targets the legacy streaming path, used whenever a
transfer is not a whole-SSTable zero-copy send.
- Raise the send window to 2 MiB. The path sized its window from the Netty
channel high water mark, 64 KiB. That is far below the bandwidth-delay product
of a high-latency inter-node link, so the sender sat waiting for the channel to
drain and throughput was latency-bound, not bandwidth-bound.
- Flush once per file, not once per section. flush() parks until every
in-flight byte reaches the network, so each section boundary emptied the send
pipe and cost a round-trip.
- Raise the network chunk to 128 KiB, configurable. Fewer syscalls, fewer
network frames and fewer compression calls per file. 128 KiB is the largest
buffer still served from the networking pool.
- Refuse a chunk larger than the window. That pairing made the sender drain
the pipe to half the window before every chunk, which is the latency the window
exists to remove.
- Add a read-ahead buffer to the file reader, so a small read no longer stalls
the send. The reader runs on its own thread and stays up to stream_read_ahead
bytes in front, so the disk keeps working while the send window drains instead
of going idle exactly when the network is about to want more.
- Send compressed SSTable sections with sendfile. This is on the sender, not
the receiver: the sending node hands the kernel the file range and the bytes
never enter the process.
- Add a direct IO read path for encrypted streaming. sendfile cannot apply
over TLS, because TLS encrypts in user space. Streamed bytes are read once and
never wanted again, so stream_disk_access_mode: direct keeps them from evicting
what the read path needs.
How the buffer caused excess allocation
The legacy writer took a pooled off-heap buffer for every network chunk, read
the chunk into it, and handed it to Netty, which released it only after the
bytes reached the socket. At the old 64 KiB chunk that is one acquire, one copy
and one release per 64 KiB, so a 1 GiB SSTable cost roughly 16,000 of each. On
the compressed path those bytes were already in wire form, CRCs included, so
the buffer and the copy bought nothing. Handing the file range to sendfile
removes the buffer, the copy and the buffer-pool pressure from that path
entirely.
I'll get some numbers this week hopefully.
https://github.com/rustyrazorblade/cassandra/tree/19979-streaming-slowpath
was (Author: rustyrazorblade):
I've got a WIP patch for this. Going to get some numbers from a cluster to back
it all up next. I've added quite a few tests to this to ensure there's no
regressions, but I'll still put it through the wringer on real clusters.
Note: The branch only targets the legacy streaming path, used whenever a
transfer is not a whole-SSTable zero-copy send.
- Raise the send window to 2 MiB. The path sized its window from the Netty
channel high water mark, 64 KiB. That is far below the bandwidth-delay product
of a high-latency inter-node link, so the sender sat waiting for the channel to
drain and throughput was latency-bound, not bandwidth-bound.
- Flush once per file, not once per section. flush() parks until every
in-flight byte reaches the network, so each section boundary emptied the send
pipe and cost a round-trip.
- Raise the network chunk to 128 KiB, configurable. Fewer syscalls, fewer
network frames and fewer compression calls per file. 128 KiB is the largest
buffer still served from the networking pool.
- Refuse a chunk larger than the window. That pairing made the sender drain the
pipe to half the window before every chunk, which is the latency the window
exists to remove.
- Add a read-ahead buffer to the file reader, so a small read no longer stalls
the send. The reader runs on its own thread and stays up to stream_read_ahead
bytes in front, so the disk keeps working while the send window drains instead
of going idle exactly when the network is about to want more.
- Send compressed SSTable sections with sendfile. This is on the sender, not
the receiver: the sending node hands the kernel the file range and the bytes
never enter the process.
- Add a direct IO read path for encrypted streaming. sendfile cannot apply over
TLS, because TLS encrypts in user space. Streamed bytes are read once and never
wanted again, so stream_disk_access_mode: direct keeps them from evicting what
the read path needs.
How the buffer caused excess allocation
The legacy writer took a pooled off-heap buffer for every network chunk, read
the chunk into it, and handed it to Netty, which released it only after the
bytes reached the socket. At the old 64 KiB chunk that is one acquire, one copy
and one release per 64 KiB, so a 1 GiB SSTable cost roughly 16,000 of each. On
the compressed path those bytes were already in wire form, CRCs included, so
the buffer and the copy bought nothing. Handing the file range to sendfile
removes the buffer, the copy and the buffer-pool pressure from that path
entirely.
I'll get some numbers this week hopefully.
> Optimize streaming slow path
> ----------------------------
>
> Key: CASSANDRA-19979
> URL: https://issues.apache.org/jira/browse/CASSANDRA-19979
> Project: Apache Cassandra
> Issue Type: Improvement
> Reporter: Jon Haddad
> Assignee: Jon Haddad
> Priority: Normal
> Attachments: image-2024-10-04-12-40-26-727.png
>
>
> CASSANDRA-15452 is introducing an internal buffer to compaction in order to
> increase throughput while reducing IOPS. We do the same thing with our
> streaming slow path, although it's not optimal. There's a common
> misconception that the overhead comes from serde overhead, but I've found on
> a lot of devices the overhead is due to our read patterns. This is most
> commonly found on non-NVMe drives, especially disaggregated storage such as
> EBS where the latency is higher and more variable.
> Attached is a perf profile showing the cost of streaming is dominated by
> pread. The team I was working with was seeing they could stream only 12MB
> per streaming session. Reducing the number of read operations by using
> larger buffered reads should improve this by at least 3-5x on some systems,
> as well as reduce CPU overhead from reduced system calls.
> I think we need to do a few things:
> * Use a larger internal buffer on disk reads.
> * Buffer writes to the network. Writing constant small values to the
> network has a very high latency cost, we'd be better off flushing larger
> values more often
> * Move the blocking network writing to a separate thread. We don't need to
> wait on the network transfer in order to read more data off disk. Once we
> improve the internal buffer on reads I think we'll see this as the next
> problem so let's tackle it now. ExecutorService.newSingleThreadExecutor()
> would work well for this.
>
>
>
>
>
>
>
>
> !image-2024-10-04-12-40-26-727.png|width=912,height=592!
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
