Like macOS we have similar issue on Linux. For TCP socket the send
buffer size is 2626560 bytes (~2.5 MiB) and we get good performance.
However for unix socket the default and maximum buffer size is 212992
bytes (208 KiB) and we see poor performance when using one NBD
connection, up to 4 times slower than macOS on the same machine.
Tracing shows that for every 2 MiB payload (qemu uses 2 MiB io size), we
do 1 recvmsg call with TCP socket, and 10 recvmsg calls with unix
socket.
Fixing this issue requires changing the maximum send buffer size (the
receive buffer size is ignored). This can be done using:
$ cat /etc/sysctl.d/net-mem-max.conf
net.core.wmem_max = 2097152
$ sudo sysctl -p /etc/sysctl.d/net-mem-max.conf
With this we can set the socket buffer size to 2 MiB. With the defaults
the value requested by qemu is clipped to the maximum size and has no
effect.
I tested on 2 machines:
- Fedora 42 VM on MacBook Pro M2 Max
- Dell PowerEdge R640 (Intel(R) Xeon(R) Gold 6230 CPU @ 2.10GHz)
On the older Dell machine we see very little improvement, up to 1.03
higher throughput. On the M2 machine we see up to 2.67 times higher
throughput. The following results are from the M2 machine.
Reading from qemu-nbd with qemu-img convert. In this test buffer size of
4m is optimal (2.28 times faster).
| buffer size | time | user | system |
|-------------|---------|---------|---------|
| default | 4.292 | 0.243 | 1.604 |
| 524288 | 2.167 | 0.058 | 1.288 |
| 1048576 | 2.041 | 0.060 | 1.238 |
| 2097152 | 1.884 | 0.060 | 1.191 |
| 4194304 | 1.881 | 0.054 | 1.196 |
Writing to qemu-nbd with qemu-img convert. In this test buffer size of
1m is optimal (2.67 times faster).
| buffer size | time | user | system |
|-------------|---------|---------|---------|
| default | 3.113 | 0.334 | 1.094 |
| 524288 | 1.173 | 0.179 | 0.654 |
| 1048576 | 1.164 | 0.164 | 0.670 |
| 2097152 | 1.227 | 0.197 | 0.663 |
| 4194304 | 1.227 | 0.198 | 0.666 |
Computing a blkhash with nbdcopy. In this test buffer size of 512k is
optimal (1.19 times faster).
| buffer size | time | user | system |
|-------------|---------|---------|---------|
| default | 2.140 | 4.483 | 2.681 |
| 524288 | 1.794 | 4.467 | 2.572 |
| 1048576 | 1.807 | 4.447 | 2.644 |
| 2097152 | 1.822 | 4.461 | 2.698 |
| 4194304 | 1.827 | 4.465 | 2.700 |
Computing a blkhash with blksum. In this test buffer size of 4m is
optimal (2.65 times faster).
| buffer size | time | user | system |
|-------------|---------|---------|---------|
| default | 3.582 | 4.595 | 2.392 |
| 524288 | 1.499 | 4.384 | 1.482 |
| 1048576 | 1.377 | 4.381 | 1.345 |
| 2097152 | 1.388 | 4.389 | 1.354 |
| 4194304 | 1.352 | 4.395 | 1.302 |
Signed-off-by: Nir Soffer <nir...@gmail.com>
---
io/channel-socket.c | 9 +++++----
1 file changed, 5 insertions(+), 4 deletions(-)
diff --git a/io/channel-socket.c b/io/channel-socket.c
index 06901ab694..f2974fab74 100644
--- a/io/channel-socket.c
+++ b/io/channel-socket.c
@@ -39,12 +39,13 @@
#define SOCKET_MAX_FDS 16
/*
- * Testing shows that 2m send buffer gives best throuput and lowest cpu usage.
- * Changing the receive buffer size has no effect on performance.
+ * Testing shows that 2m send buffer is optimal. Changing the receive buffer
+ * size has no effect on performance.
+ * On Linux we need to increase net.core.wmem_max to make this effective.
*/
-#ifdef __APPLE__
+#if defined(__APPLE__) || defined(__linux__)
#define UNIX_STREAM_SOCKET_SEND_BUFFER_SIZE (2 * MiB)
-#endif /* __APPLE__ */
+#endif
static void qio_channel_socket_set_buffers(QIOChannelSocket *ioc)
{
--
2.39.5 (Apple Git-154)
