On Tue, Jun 27, 2023 at 6:54 PM Xuan Zhuo <[email protected]> wrote:
>
> On Tue, 27 Jun 2023 17:00:06 +0800, Jason Wang <[email protected]> wrote:
> > On Tue, Jun 27, 2023 at 4:28 PM Xuan Zhuo <[email protected]>
> > wrote:
> > >
> > >
> > > Thanks Parav for pointing it out. We may have some gaps on the case.
> > >
> > > Let me introduce our case, which I think it is simple and should be easy
> > > to
> > > understand.
> > >
> > > First, the user (customer) purchased a bare metal machine.
> > >
> > > ## Bare metal machine
> > >
> > > Let me briefly explain the characteristics of a bare metal machine. It is
> > > not a
> > > virtual machine, it is a physical machine, and the difference between it
> > > and a
> > > general physical machine is that its PCI is connected to a device similar
> > > to a
> > > DPU. This DPU provides devices such as virtio-blk/net to the host through
> > > PCI.
> > > These devices are managed by the vendor, and must be created and purchased
> > > on the vendor's management platform.
> > >
> > > ## DPU
> > >
> > > There is a software implementation in the DPU, which will respond to PCI
> > > operations. But as mentioned above, resources such as network cards must
> > > be
> > > purchased and created before they can exist. So users can create VF,
> > > which is
> > > just a pci-level operation, but there may not be a corresponding backend.
> > >
> > > ## Management Platform
> > >
> > > The creation and configuration of devices is realized on the management
> > > platform.
> > >
> > > After the user completed the purchase on the management platform (this is
> > > an
> > > independent platform provided by the vendor and has nothing to do with
> > > virtio), then there will be a corresponding device implementation in the
> > > DPU.
> > > This includes some user configurations, available bandwidth resources and
> > > other
> > > information.
> > >
> > > ## Usage
> > >
> > > Since the user is directly on the HOST, the user can create VMs,
> > > passthrough PF
> > > or VF into the VM. Or users can create a large number of dockers, all of
> > > which
> > > use a separate virtio-net device for performance.
> > >
> > > The reason why users use vf is that we need to use a large number of
> > > virtio-net
> > > devices. This number reaches 1k+.
> > >
> > > Based on this scenario, we need to bind vf to the backend device.
> > > Because, we
> > > cannot automatically complete the creation of the virtio-net backend
> > > device when
> > > the user creates a vf.
> > >
> > > ## Migration
> > >
> > > In addition, let's consider another scenario of migration. If a vm is
> > > migrated
> > > from another host, of course its corresponding virtio device is also
> > > migrated to
> > > the DPU. At this time, our newly created vf can only be used by the vm
> > > after it
> > > is bound to the migrated device. We do not want this vf to be a brand new
> > > device.
> > >
> > > ## Abstraction
> > >
> > > So, this is how I understand the process of creating vf:
> > >
> > > 1. Create a PCI VF, at this time there may be no backend virtio device,
> > > or there
> > > is only a default backend. It does not fully meet our expectations.
> > > 2. Create device or migrate device
> > > 3. Bind the backend virtio device to the vf
> >
> > 3) should come before 2)?
> >
> > Who is going to do 3) btw, is it the user? If yes, for example, if a
> > user wants another 4 queue virtio-net devices, after purchase, how
> > does the user know its id?
>
> Got the id from the management platform.
So it can do the binding via that management platform which this
became a cloud vendor specific interface.
>
> >
> > >
> > > In most scenarios, the first step may be enough. We can make some
> > > fine-tuning on
> > > this default device, such as modifying its mac. In the future, we can use
> > > admin
> > > queue to modify its msix vector and other configurations.
> > >
> > > But we should allow, we bind a backend virtio device to a certain vf.
> > > This is
> > > useful for live migration and virtio devices with special configurations.
> >
> > All of these could be addressed if a dynamic provisioning model is
> > implemented (SIOV or transport virtqueue). Trying to have a workaround
> > in SR-IOV might be tricky.
>
>
> SR-IOV vf is native PCI device, this is the advancement.
The problem is that it doesn't support flexible provisioning, e.g
create and destroy a single VF.
>
>
> >
> > >
> > > The design of virtio itself is two layers, and virtio should allow
> > > switching the
> > > transport layer by nature. This is our advantage.
> >
> > Is it not switching the transport layer but about binding/unbinding
> > vitio devices to VF?
>
> YES.
>
> >
> > Is a new capability or similar admin cmd sufficient in this case?
>
> All is ok.
>
>
> >
> > struct virtio_pci_bind_cap {
> > struct virtio_pci_cap cap;
> > u16 bind; // virtio_device_id
> > u16 unbind; // virtio_device_id
> > };
>
> You mean that the "bind" or "unbind" is writeable?
Yes.
Thanks
>
> Thanks.
>
> >
> > Thanks
> >
> > >
> > > ## About the identity
> > >
> > > In this patch, I used a vendor's id. The purpose of this is that I hope to
> > > be compatible with all devices. In the network scenario, it is actually
> > > possible
> > > to use a mac.
> > >
> > > Perhaps, introduce a standard id for each device/driver, I think this may
> > > be
> > > more in line with the habit of virtio spec.
> > >
> > > Thanks.
> > >
> > > On Mon, 26 Jun 2023 14:22:10 +0800, Xuan Zhuo
> > > <[email protected]> wrote:
> > > > The VFs of the SR-IOV are created by the user inside the guest OS, so
> > > > the virtio
> > > > devices don't know about these VFs. Because each VF may be assigned a
> > > > different role
> > > > by the user, the virtio device can not choose one VF to bind random.
> > > > So only the user knows how to bind the virtio devices to the VFs.
> > > > On the other hand, generally the virtio devices are not created by the
> > > > user
> > > > inside the guest OS. This requires some management platform to
> > > > participate.
> > > >
> > > > So the usage of this command:
> > > > 1. The user purchases a virtio network card on the management platform,
> > > > and sets the ip, queue number, etc. The user obtains the identity of
> > > > the network card.
> > > > 2. The user creates a VF with echo 8 > sriov_numvfs
> > > > 3. The user binds the net crad to a VF with identity through the command
> > > > of the patch
> > > >
> > > > Signed-off-by: Xuan Zhuo <[email protected]>
> > > > ---
> > > > admin.tex | 41 ++++++++++++++++++++++++++++++++++++++++-
> > > > 1 file changed, 40 insertions(+), 1 deletion(-)
> > > >
> > > > diff --git a/admin.tex b/admin.tex
> > > > index 2efd4d7..64d0667 100644
> > > > --- a/admin.tex
> > > > +++ b/admin.tex
> > > > @@ -115,7 +115,8 @@ \subsection{Group administration
> > > > commands}\label{sec:Basic Facilities of a Virti
> > > > \hline \hline
> > > > 0x0000 & VIRTIO_ADMIN_CMD_LIST_QUERY & Provides to driver list of
> > > > commands supported for this group type \\
> > > > 0x0001 & VIRTIO_ADMIN_CMD_LIST_USE & Provides to device list of
> > > > commands used for this group type \\
> > > > -0x0002 - 0x7FFF & - & Commands using \field{struct virtio_admin_cmd}
> > > > \\
> > > > +0x0002 & VIRTIO_ADMIN_CMD_BIND_DEVICE & Bind the device to one group
> > > > member \\
> > > > +0x0003 - 0x7FFF & - & Commands using \field{struct virtio_admin_cmd}
> > > > \\
> > > > \hline
> > > > 0x8000 - 0xFFFF & - & Reserved for future commands (possibly using a
> > > > different structure) \\
> > > > \hline
> > > > @@ -429,6 +430,44 @@ \subsection{Group administration
> > > > commands}\label{sec:Basic Facilities of a Virti
> > > > \field{VF Enable} refer to registers within the SR-IOV Extended
> > > > Capability as specified by \hyperref[intro:PCIe]{[PCIe]}.
> > > >
> > > > +\subsubsection{Bind the device for member}
> > > > +
> > > > +The VFs of the SR-IOV are created by the user inside the guest OS, so
> > > > the virtio
> > > > +devices don't know about these VFs. Because each VF may be assigned a
> > > > different role
> > > > +by the user, the virtio device can not choose one VF to bind random.
> > > > +So only the user knows how to bind the virtio devices to the VFs.
> > > > +On the other hand, generally the virtio devices are not created by the
> > > > user
> > > > +inside the guest OS. This requires some management platform to
> > > > participate.
> > > > +
> > > > +So we introduce a new admin queue command to bind the VFs and the
> > > > virtio
> > > > +devices.
> > > > +
> > > > +\begin{lstlisting}
> > > > +struct virtio_admin_cmd_bind {
> > > > + u64 identity;
> > > > +};
> > > > +\end{lstlisting}
> > > > +
> > > > +The user got the \field{identity} from the management platform, that
> > > > is not
> > > > +included by this spec.
> > > > +
> > > > +\drivernormative{\paragraph}{Group administration commands}{Basic
> > > > Facilities of a Virtio Device / Device groups / Group administration
> > > > commands / Bind the device for member}
> > > > +
> > > > +VIRTIO_ADMIN_CMD_BIND_DEVICE requires that the \field{group_member_id}
> > > > MUST be set.
> > > > +
> > > > +The \field{identity} is passed by the user. It is the identity of the
> > > > virtio
> > > > +device.
> > > > +
> > > > +\devicenormative{\paragraph}{Group administration commands}{Basic
> > > > Facilities of a Virtio Device / Device groups / Group administration
> > > > commands / Bind the device for member}
> > > > +
> > > > +Every device MUST have one unique \field{identity} in the host.
> > > > +
> > > > +If the PF device can not find the device by the \field{identity},
> > > > +the \field{status} MUST be set to VIRTIO_ADMIN_STATUS_EINVAL.
> > > > +
> > > > +If the device is found by the \field{identity}, the device MUST work
> > > > as the
> > > > +device of this group member specified by the \field{group_member_id}.
> > > > +
> > > > \section{Administration Virtqueues}\label{sec:Basic Facilities of a
> > > > Virtio Device / Administration Virtqueues}
> > > >
> > > > An administration virtqueue of an owner device is used to submit
> > > > --
> > > > 2.32.0.3.g01195cf9f
> > > >
> > > >
> > > > This publicly archived list offers a means to provide input to the
> > > > OASIS Virtual I/O Device (VIRTIO) TC.
> > > >
> > > > In order to verify user consent to the Feedback License terms and
> > > > to minimize spam in the list archive, subscription is required
> > > > before posting.
> > > >
> > > > Subscribe: [email protected]
> > > > Unsubscribe: [email protected]
> > > > List help: [email protected]
> > > > List archive: https://lists.oasis-open.org/archives/virtio-comment/
> > > > Feedback License:
> > > > https://www.oasis-open.org/who/ipr/feedback_license.pdf
> > > > List Guidelines:
> > > > https://www.oasis-open.org/policies-guidelines/mailing-lists
> > > > Committee: https://www.oasis-open.org/committees/virtio/
> > > > Join OASIS: https://www.oasis-open.org/join/
> > > >
> > >
> >
>
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
