On 27.03.25 16:55, Stefan Hajnoczi wrote:
On Tue, Mar 25, 2025 at 05:06:54PM +0100, Hanna Czenczek wrote:
FUSE allows creating multiple request queues by "cloning" /dev/fuse FDs
(via open("/dev/fuse") + ioctl(FUSE_DEV_IOC_CLONE)).
We can use this to implement multi-threading.
Note that the interface presented here differs from the multi-queue
interface of virtio-blk: The latter maps virtqueues to iothreads, which
allows processing multiple virtqueues in a single iothread. The
equivalent (processing multiple FDs in a single iothread) would not make
sense for FUSE because those FDs are used in a round-robin fashion by
the FUSE kernel driver. Putting two of them into a single iothread will
just create a bottleneck.
This text might be outdated. virtio-blk's new iothread-vq-mapping
parameter provides the "array of iothreads" mentioned below and a way to
assign virtqueues to those IOThreads.
Ah, yes. The difference is still that with FUSE, there is no such
assignment, because it wouldn’t make sense. But I can change s/maps
virtqueues/allows mapping virtqueues/, and s/differs from/is only a
subset of/, if that’s alright.
Therefore, all we need is an array of iothreads, and we will create one
"queue" (FD) per thread.
These are the benchmark results when using four threads (compared to a
single thread); note that fio still only uses a single job, but
performance can still be improved because of said round-robin usage for
the queues. (Not in the sync case, though, in which case I guess it
just adds overhead.)
Interesting. FUSE-over-io_uring seems to be different from
FUSE_DEV_IOC_CLONE here. It doesn't do round-robin. It uses CPU affinity
instead, handing requests to the io_uring context associated with the
current CPU when possible.
Do you think that should have implications for the QAPI interface?
[...]
qapi/block-export.json | 8 +-
block/export/fuse.c | 214 +++++++++++++++++++++++++++++++++--------
2 files changed, 179 insertions(+), 43 deletions(-)
diff --git a/qapi/block-export.json b/qapi/block-export.json
index c783e01a53..0bdd5992eb 100644
--- a/qapi/block-export.json
+++ b/qapi/block-export.json
@@ -179,12 +179,18 @@
# mount the export with allow_other, and if that fails, try again
# without. (since 6.1; default: auto)
#
+# @iothreads: Enables multi-threading: Handle requests in each of the
+# given iothreads (instead of the block device's iothread, or the
+# export's "main" iothread). For this, the FUSE FD is duplicated so
+# there is one FD per iothread. (since 10.1)
This option isn't FUSE-specific but FUSE is the first export type to
support it. Please add it to BlockExportOptions instead and refuse
export creation when the export type only supports 1 IOThread.
Makes sense. I’ll try to go with what Kevin suggested, i.e. have
@iothread be an alternate type.
Hanna
Eric: Are you interested in implementing support for multiple IOThreads
in the NBD export? I remember some time ago we talked about NBD
multi-conn support, although maybe that was for the client rather than
the server.
