A QEMU built-in VIRTIO device will also call virtio_add_queue() for
the maximum number of virtqueues.
I'm not sure what the concern is about adding as few virtqueues as possible?
If the front-end's implementation is inefficient, then it should be
optimized so that untouched virtqueues don't consume resources. I
don't see the need to try to add a special message to vhost-user to
try to reduce the number of virtqueues.
Stefan
On Fri, 8 Sept 2023 at 08:03, Alex Bennée <alex.ben...@linaro.org> wrote:
>
>
> Stefan Hajnoczi <stefa...@gmail.com> writes:
>
> > On Fri, 8 Sept 2023 at 02:43, Alex Bennée <alex.ben...@linaro.org> wrote:
> >>
> >>
> >> Stefan Hajnoczi <stefa...@redhat.com> writes:
> >>
> >> > On Tue, Sep 05, 2023 at 10:34:11AM +0100, Alex Bennée wrote:
> >> >>
> >> >> Albert Esteve <aest...@redhat.com> writes:
> >> >>
> >> >> > This looks great! Thanks for this proposal.
> >> >> >
> >> >> > On Fri, Sep 1, 2023 at 1:00 PM Alex Bennée <alex.ben...@linaro.org>
> >> >> > wrote:
> >> >> >
> >> >> > Currently QEMU has to know some details about the VirtIO device
> >> >> > supported by a vhost-user daemon to be able to setup the guest. This
> >> >> > makes it hard for QEMU to add support for additional vhost-user
> >> >> > daemons without adding specific stubs for each additional VirtIO
> >> >> > device.
> >> >> >
> >> >> > This patch suggests a new feature flag (VHOST_USER_PROTOCOL_F_PROBE)
> >> >> > which the back-end can advertise which allows a probe message to be
> >> >> > sent to get all the details QEMU needs to know in one message.
> >> >> >
> >> >> > Together with the existing features VHOST_USER_PROTOCOL_F_STATUS and
> >> >> > VHOST_USER_PROTOCOL_F_CONFIG we can create "standalone" vhost-user
> >> >> > daemons which are capable of handling all aspects of the VirtIO
> >> >> > transactions with only a generic stub on the QEMU side. These daemons
> >> >> > can also be used without QEMU in situations where there isn't a full
> >> >> > VMM managing their setup.
> >> >> >
> >> >> > Signed-off-by: Alex Bennée <alex.ben...@linaro.org>
> >> >> >
> >> >> > ---
> >> >> > v2
> >> >> > - dropped F_STANDALONE in favour of F_PROBE
> >> >> > - split probe details across several messages
> >> >> > - probe messages don't automatically imply a standalone daemon
> >> >> > - add wording where probe details interact (F_MQ/F_CONFIG)
> >> >> > - define VMM and make clear QEMU is only one of many potential VMMs
> >> >> > - reword commit message
> >> >> > ---
> >> >> > docs/interop/vhost-user.rst | 90
> >> >> > ++++++++++++++++++++++++++++++++-----
> >> >> > hw/virtio/vhost-user.c | 8 ++++
> >> >> > 2 files changed, 88 insertions(+), 10 deletions(-)
> >> >> >
> >> >> > diff --git a/docs/interop/vhost-user.rst
> >> >> > b/docs/interop/vhost-user.rst
> >> >> > index 5a070adbc1..ba3b5e07b7 100644
> >> >> > --- a/docs/interop/vhost-user.rst
> >> >> > +++ b/docs/interop/vhost-user.rst
> >> >> > @@ -7,6 +7,7 @@ Vhost-user Protocol
> >> >> > ..
> >> >> > Copyright 2014 Virtual Open Systems Sarl.
> >> >> > Copyright 2019 Intel Corporation
> >> >> > + Copyright 2023 Linaro Ltd
> >> >> > Licence: This work is licensed under the terms of the GNU GPL,
> >> >> > version 2 or later. See the COPYING file in the top-level
> >> >> > directory.
> >> >> > @@ -27,17 +28,31 @@ The protocol defines 2 sides of the
> >> >> > communication, *front-end* and
> >> >> > *back-end*. The *front-end* is the application that shares its
> >> >> > virtqueues, in
> >> >> > our case QEMU. The *back-end* is the consumer of the virtqueues.
> >> >> >
> >> >> > -In the current implementation QEMU is the *front-end*, and the
> >> >> > *back-end*
> >> >> > -is the external process consuming the virtio queues, for example a
> >> >> > -software Ethernet switch running in user space, such as Snabbswitch,
> >> >> > -or a block device back-end processing read & write to a virtual
> >> >> > -disk. In order to facilitate interoperability between various
> >> >> > back-end
> >> >> > -implementations, it is recommended to follow the :ref:`Backend
> >> >> > program
> >> >> > -conventions <backend_conventions>`.
> >> >> > +In the current implementation a Virtual Machine Manager (VMM) such
> >> >> > as
> >> >> > +QEMU is the *front-end*, and the *back-end* is the external process
> >> >> > +consuming the virtio queues, for example a software Ethernet switch
> >> >> > +running in user space, such as Snabbswitch, or a block device
> >> >> > back-end
> >> >> > +processing read & write to a virtual disk. In order to facilitate
> >> >> > +interoperability between various back-end implementations, it is
> >> >> > +recommended to follow the :ref:`Backend program conventions
> >> >> > +<backend_conventions>`.
> >> >> >
> >> >> > The *front-end* and *back-end* can be either a client (i.e.
> >> >> > connecting) or
> >> >> > server (listening) in the socket communication.
> >> >> >
> >> >> > +Probing device details
> >> >> > +----------------------
> >> >> > +
> >> >> > +Traditionally the vhost-user daemon *back-end* shares configuration
> >> >> > +responsibilities with the VMM *front-end* which needs to know
> >> >> > certain
> >> >> > +key bits of information about the device. This means the VMM needs
> >> >> > to
> >> >> > +define at least a minimal stub for each VirtIO device it wants to
> >> >> > +support. If the daemon supports the right set of protocol features
> >> >> > the
> >> >> > +VMM can probe the daemon for the information it needs to setup the
> >> >> > +device. See :ref:`Probing features for standalone daemons
> >> >> > +<probing_features>` for more details.
> >> >> > +
> >> >> > +
> >> >> > Support for platforms other than Linux
> >> >> > --------------------------------------
> >> >> >
> >> >> > @@ -316,6 +331,7 @@ replies. Here is a list of the ones that do:
> >> >> > * ``VHOST_USER_GET_VRING_BASE``
> >> >> > * ``VHOST_USER_SET_LOG_BASE`` (if
> >> >> > ``VHOST_USER_PROTOCOL_F_LOG_SHMFD``)
> >> >> > * ``VHOST_USER_GET_INFLIGHT_FD`` (if
> >> >> > ``VHOST_USER_PROTOCOL_F_INFLIGHT_SHMFD``)
> >> >> > +* ``VHOST_USER_GET_BACKEND_SPECS`` (if
> >> >> > ``VHOST_USER_PROTOCOL_F_STANDALONE``)
> >> >> >
> >> >> > .. seealso::
> >> >> >
> >> >> > @@ -396,9 +412,10 @@ must support changing some configuration
> >> >> > aspects on the fly.
> >> >> > Multiple queue support
> >> >> > ----------------------
> >> >> >
> >> >> > -Many devices have a fixed number of virtqueues. In this case the
> >> >> > front-end
> >> >> > -already knows the number of available virtqueues without
> >> >> > communicating with the
> >> >> > -back-end.
> >> >> > +Many devices have a fixed number of virtqueues. In this case the
> >> >> > +*front-end* usually already knows the number of available virtqueues
> >> >> > +without communicating with the back-end. For standalone daemons this
> >> >> > +number can be can be probed with the ``VHOST_USER_GET_MIN_VQ``
> >> >> > message.
> >> >> >
> >> >> > Some devices do not have a fixed number of virtqueues. Instead the
> >> >> > maximum
> >> >> > number of virtqueues is chosen by the back-end. The number can
> >> >> > depend on host
> >> >> > @@ -885,6 +902,23 @@ Protocol features
> >> >> > #define VHOST_USER_PROTOCOL_F_CONFIGURE_MEM_SLOTS 15
> >> >> > #define VHOST_USER_PROTOCOL_F_STATUS 16
> >> >> > #define VHOST_USER_PROTOCOL_F_XEN_MMAP 17
> >> >> > + #define VHOST_USER_PROTOCOL_F_PROBE 18
> >> >> > +
> >> >> > +.. _probing_features:
> >> >> > +
> >> >> > +Probing features for standalone daemons
> >> >> > +---------------------------------------
> >> >> > +
> >> >> > +The protocol feature ``VHOST_USER_PROTOCOL_F_PROBE`` enables a
> >> >> > number
> >> >> > +of additional messages which allow the *front-end* to probe details
> >> >> > +about the VirtIO device from the *back-end*. However for a
> >> >> > *back-end*
> >> >> > +to be described as standalone it must also support:
> >> >> > +
> >> >> > + * ``VHOST_USER_PROTOCOL_F_STATUS``
> >> >> > + * ``VHOST_USER_PROTOCOL_F_CONFIG`` (if there is a config space)
> >> >> > +
> >> >> > +which are required to ensure the *back-end* daemon can operate
> >> >> > +without the *front-end* managing some aspects of its configuration.
> >> >> >
> >> >> > Front-end message types
> >> >> > -----------------------
> >> >> > @@ -1440,6 +1474,42 @@ Front-end message types
> >> >> > query the back-end for its device status as defined in the Virtio
> >> >> > specification.
> >> >> >
> >> >> > +``VHOST_USER_GET_DEVICE_ID``
> >> >> > + :id: 41
> >> >> > + :request payload: N/A
> >> >> > + :reply payload: ``u32``
> >> >> > +
> >> >> > + When the ``VHOST_USER_PROTOCOL_F_PROBE`` protocol feature has been
> >> >> > + successfully negotiated, this message is submitted by the
> >> >> > front-end
> >> >> > + to query what VirtIO device the back-end support. This is intended
> >> >> > + to remove the need for the front-end to know ahead of time what
> >> >> > the
> >> >> > + VirtIO device the backend emulates is.
> >> >> > +
> >> >> > +``VHOST_USER_GET_CONFIG_SIZE``
> >> >> > + :id: 42
> >> >> > + :request payload: N/A
> >> >> > + :reply payload: ``u32``
> >> >> > +
> >> >> > + When the ``VHOST_USER_PROTOCOL_F_PROBE`` protocol feature has been
> >> >> > + successfully negotiated, this message is submitted by the
> >> >> > front-end
> >> >> > + to query the size of the VirtIO device's config space. This is
> >> >> > + intended to remove the need for the front-end to know ahead of
> >> >> > time
> >> >> > + what the size is. Replying with 0 when
> >> >> > + ``VHOST_USER_PROTOCOL_F_CONFIG`` has been negotiated would
> >> >> > indicate
> >> >> > + an bug.
> >> >> > +
> >> >> > +``VHOST_USER_GET_MIN_VQ``
> >> >> > + :id: 43
> >> >> > + :request payload: N/A
> >> >> > + :reply payload: ``u32``
> >> >> > +
> >> >> > + When the ``VHOST_USER_PROTOCOL_F_PROBE`` protocol feature has been
> >> >> > + successfully negotiated, this message is submitted by the
> >> >> > front-end to
> >> >> > + query minimum number of VQ's required to support the device. A
> >> >> > + device may support more than this number of VQ's if it advertises
> >> >> > + the ``VHOST_USER_PROTOCOL_F_MQ`` protocol feature. Reporting a
> >> >> > + number greater than the result of ``VHOST_USER_GET_QUEUE_NUM``
> >> >> > would
> >> >> > + indicate a bug.
> >> >> >
> >> >> > Maybe I lack some background, but not sure what min_vq is here?
> >> >>
> >> >> There will be a minimum number of queues you need to support the device.
> >> >> For example the virtio-sound spec specifies you need four queues:
> >> >> control, event, tx, rx
> >> >
> >> > I don't understand why the front-end needs to know that? The backend
> >> > already reports the number of queues and not all of them need to be
> >> > initialized by the driver.
> >>
> >> But how many don't need to be initialised? We can't just skip:
> >>
> >> /* Allocate queues */
> >> vub->vqs = g_ptr_array_sized_new(vub->num_vqs);
> >> for (int i = 0; i < vub->num_vqs; i++) {
> >> g_ptr_array_add(vub->vqs,
> >> virtio_add_queue(vdev, vub->vq_size,
> >> vub_handle_output));
> >> }
> >>
> >> Or are you saying just require probe-able backends to support
> >> VHOST_USER_PROTOCOL_F_MQ and have it always report the minimmum number
> >> of queues if it is not a MQ capable device?
> >
> > The front-end should prepare to allow the maximum number of virtqueues
> > returned by VHOST_USER_GET_QUEUE_NUM (VHOST_USER_PROTOCOL_F_MQ).
> >
> > VIRTIO Transports have a way to query the maximum number of queues but
> > not a way to query the minimum number of queues. Why is the minimum
> > necessary?
>
> It seems excessive to automatically create the maximum number of VQs. I
> guess for backends that don't support the MQ feature (i.e. a variable
> number of VQs) we could just say VHOST_USER_GET_QUEUE_NUM == min. But
> now we are overloading a different message originally added for
> something else.
>
> --
> Alex Bennée
> Virtualisation Tech Lead @ Linaro
