On Fri, Apr 14, 2023 at 10:25 PM Ferruh Yigit <ferruh.yi...@amd.com> wrote: > > On 4/14/2023 1:06 PM, Maxime Coquelin wrote: > > > > > > On 4/14/23 12:48, Ferruh Yigit wrote: > >> On 4/13/2023 8:59 AM, Maxime Coquelin wrote: > >>> Hi, > >>> > >>> On 4/13/23 09:08, Xia, Chenbo wrote: > >>>>> -----Original Message----- > >>>>> From: Morten Brørup <m...@smartsharesystems.com> > >>>>> Sent: Thursday, April 13, 2023 3:41 AM > >>>>> To: Maxime Coquelin <maxime.coque...@redhat.com>; Ferruh Yigit > >>>>> <ferruh.yi...@amd.com>; dev@dpdk.org; david.march...@redhat.com; Xia, > >>>>> Chenbo <chenbo....@intel.com>; m...@redhat.com; f...@redhat.com; > >>>>> jasow...@redhat.com; Liang, Cunming <cunming.li...@intel.com>; Xie, > >>>>> Yongji > >>>>> <xieyon...@bytedance.com>; echau...@redhat.com; epere...@redhat.com; > >>>>> amore...@redhat.com > >>>>> Subject: RE: [RFC 00/27] Add VDUSE support to Vhost library > >>>>> > >>>>>> From: Maxime Coquelin [mailto:maxime.coque...@redhat.com] > >>>>>> Sent: Wednesday, 12 April 2023 17.28 > >>>>>> > >>>>>> Hi Ferruh, > >>>>>> > >>>>>> On 4/12/23 13:33, Ferruh Yigit wrote: > >>>>>>> On 3/31/2023 4:42 PM, Maxime Coquelin wrote: > >>>>>>>> This series introduces a new type of backend, VDUSE, > >>>>>>>> to the Vhost library. > >>>>>>>> > >>>>>>>> VDUSE stands for vDPA device in Userspace, it enables > >>>>>>>> implementing a Virtio device in userspace and have it > >>>>>>>> attached to the Kernel vDPA bus. > >>>>>>>> > >>>>>>>> Once attached to the vDPA bus, it can be used either by > >>>>>>>> Kernel Virtio drivers, like virtio-net in our case, via > >>>>>>>> the virtio-vdpa driver. Doing that, the device is visible > >>>>>>>> to the Kernel networking stack and is exposed to userspace > >>>>>>>> as a regular netdev. > >>>>>>>> > >>>>>>>> It can also be exposed to userspace thanks to the > >>>>>>>> vhost-vdpa driver, via a vhost-vdpa chardev that can be > >>>>>>>> passed to QEMU or Virtio-user PMD. > >>>>>>>> > >>>>>>>> While VDUSE support is already available in upstream > >>>>>>>> Kernel, a couple of patches are required to support > >>>>>>>> network device type: > >>>>>>>> > >>>>>>>> https://gitlab.com/mcoquelin/linux/-/tree/vduse_networking_poc > >>>>>>>> > >>>>>>>> In order to attach the created VDUSE device to the vDPA > >>>>>>>> bus, a recent iproute2 version containing the vdpa tool is > >>>>>>>> required. > >>>>>>> > >>>>>>> Hi Maxime, > >>>>>>> > >>>>>>> Is this a replacement to the existing DPDK vDPA framework? What > >>>>>>> is the > >>>>>>> plan for long term? > >>>>>>> > >>>>>> > >>>>>> No, this is not a replacement for DPDK vDPA framework. > >>>>>> > >>>>>> We (Red Hat) don't have plans to support DPDK vDPA framework in our > >>>>>> products, but there are still contribution to DPDK vDPA by several > >>>>>> vDPA > >>>>>> hardware vendors (Intel, Nvidia, Xilinx), so I don't think it is > >>>>>> going > >>>>>> to be deprecated soon. > >>>>> > >>>>> Ferruh's question made me curious... > >>>>> > >>>>> I don't know anything about VDUSE or vDPA, and don't use any of it, so > >>>>> consider me ignorant in this area. > >>>>> > >>>>> Is VDUSE an alternative to the existing DPDK vDPA framework? What are > >>>>> the > >>>>> differences, e.g. in which cases would an application developer (or > >>>>> user) > >>>>> choose one or the other? > >>>> > >>>> Maxime should give better explanation.. but let me just explain a bit. > >>>> > >>>> Vendors have vDPA HW that support vDPA framework (most likely in their > >>>> DPU/IPU > >>>> products). This work is introducing a way to emulate a SW vDPA > >>>> device in > >>>> userspace (DPDK), and this SW vDPA device also supports vDPA framework. > >>>> > >>>> So it's not an alternative to existing DPDK vDPA framework :) > >>> > >>> Correct. > >>> > >>> When using DPDK vDPA, the datapath of a Vhost-user port is offloaded to > >>> a compatible physical NIC (i.e. a NIC that implements Virtio rings > >>> support), the control path remains the same as a regular Vhost-user > >>> port, i.e. it provides a Vhost-user unix socket to the application (like > >>> QEMU or DPDK Virtio-user PMD). > >>> > >>> When using Kernel vDPA, the datapath is also offloaded to a vDPA > >>> compatible device, and the control path is managed by the vDPA bus. > >>> It can either be consumed by a Kernel Virtio device (here Virtio-net) > >>> when using Virtio-vDPA. In this case the device is exposed as a regular > >>> netdev and, in the case of Kubernetes, can be used as primary interfaces > >>> for the pods. > >>> Or it can be exposed to user-space via Vhost-vDPA, a chardev that can be > >>> seen as an alternative to Vhost-user sockets. In this case it can for > >>> example be used by QEMU or DPDK Virtio-user PMD. In Kubernetes, it can > >>> be used as a secondary interface. > >>> > >>> Now comes VDUSE. VDUSE is a Kernel vDPA device, but instead of being a > >>> physical device where the Virtio datapath is offloaded, the Virtio > >>> datapath is offloaded to a user-space application. With this series, a > >>> DPDK application, like OVS-DPDK for instance, can create VDUSE device > >>> and expose them either as regular netdev when binding them to Kernel > >>> Virtio-net driver via Virtio-vDPA, or as Vhost-vDPA interface to be > >>> consumed by another userspace appliation like QEMU or DPDK application > >>> using Virtio-user PMD. With this solution, OVS-DPDK could serve both > >>> primary and secondary interfaces of Kubernetes pods. > >>> > >>> I hope it clarifies, I will add these information in the cover-letter > >>> for next revisions. Let me know if anything is still unclear. > >>> > >>> I did a presentation at last DPDK summit [0], maybe the diagrams will > >>> help to clarify furthermore. > >>> > >> > >> Thanks Chenbo, Maxime for clarification. > >> > >> After reading a little more (I think) I got it better, slides [0] were > >> useful. > >> > >> So this is more like a backend/handler, similar to vhost-user, although > >> it is vDPA device emulation. > >> Can you please describe more the benefit of vduse comparing to > >> vhost-user? > > > > The main benefit is that VDUSE device can be exposed as a regular > > netdev, while this is not possible with Vhost-user. > > > > Got it, thanks. I think better to highlight this in commit logs. > > And out of curiosity, > Why there is no virtio(guest) to virtio(host) communication support > (without vdpa), something like adding virtio as backend to vhost-net, is > it not needed or technical difficulties?
The main reason is that a lot of operations are not supported by virtio yet: 1) virtqueue saving, restoring 2) provisioning and management 3) address space id etc Thanks > > >> Also what is "VDUSE daemon", which is referred a few times in > >> documentation, is it another userspace implementation of the vduse? > > > > VDUSE daemon is the application that implements the VDUSE device, e.g. > > OVS-DPDK with DPDK Vhost library using this series in our case. > > > > Maxime > >> > >>>>> > >>>>> And if it is a better alternative, perhaps the documentation should > >>>>> mention that it is recommended over DPDK vDPA. Just like we started > >>>>> recommending alternatives to the KNI driver, so we could phase it out > >>>>> and > >>>>> eventually get rid of it. > >>>>> > >>>>>> > >>>>>> Regards, > >>>>>> Maxime > >>>> > >>> > >>> [0]: > >>> https://static.sched.com/hosted_files/dpdkuserspace22/9f/Open%20DPDK%20to%20containers%20networking%20with%20VDUSE.pdf > >>> > >> > > >
