Hi Stephen, > -----Original Message----- > From: Stephen Hemminger <step...@networkplumber.org> > Sent: Friday, August 14, 2020 11:00 PM > To: Xia, Chenbo <chenbo....@intel.com> > Cc: dev@dpdk.org; tho...@monjalon.net; Ding, Xuan <xuan.d...@intel.com>; > Lu, Xiuchun <xiuchun...@intel.com>; Liang, Cunming > <cunming.li...@intel.com>; Liu, Changpeng <changpeng....@intel.com>; Wang, > Zhihong <zhihong.w...@intel.com> > Subject: Re: [dpdk-dev] [RFC v1 0/2] Add device emulation support in DPDK > > On Fri, 14 Aug 2020 19:16:04 +0000 > Chenbo Xia <chenbo....@intel.com> wrote: > > > This series enables DPDK to be an alternative I/O device emulation > library of > > building virtualized devices in separate processes outside QEMU. It > introduces > > a new library (librte_vfio_user), a new device class (emudev) and one > pilot > > device provider (avf_emudev) with its backend of Ethdev PMD > (avfbe_ethdev). > > > > *librte_vfio_user* is a server implementation of VFIO-over-socket[1] > (also > > known as vfio-user) which is a protocol that allows a device to be > virtualized > > in a separate process outside of QEMU. > > > > *emudev* is a device type for emulated devices. It is up to device > provider to > > choose the transport. In avf_emudev case, it uses vfio-user as transport > > communicate with its client (e.g., QEMU). > > > > *avf_emudev* is the emudev provider of AVF which is a device > specification for > > Intel Virtual Function cross generation. It’s implemented by an AVF > emudev > > driver which offers a few APIs for avfbe_ethdev or app logic to operate. > > > > *avfbe_ethdev* is a normal ethdev PMD to supply the basic I/O as backend > data > > path of avf_emudev. One simple usage of avfbe_ethdev could be a para- > virtualized > > backend connected with network application logic. > > > > Background & Motivation > > ----------------------- > > In order to reduce the attack surface, QEMU community is disaggregating > QEMU by > > removing part of device emulation from it. The disaggregated/multi- > process QEMU > > is using VFIO-over-socket/vfio-user as the main transport mechanism to > disaggregate > > I/O services from QEMU[2]. Vfio-user essentially implements the VFIO > device model > > presented to the user process by a set of messages over a unix-domain > socket. The > > main difference between application using vfio-user and application > using vfio > > kernel module is that device manipulation is based on socket messages > for vfio-user > > but system calls for vfio kernel module. The vfio-user devices consist > of a generic > > VFIO device type, living in QEMU, which is called the client[3], and the > core device > > implementation (emulated device), living outside of QEMU, which is > called the server. > > > > With the introduction and support of vfio-user in QEMU, QEMU is > explicitly adding > > support for external emulated device and data path. We are trying to > leverage that > > and introducing vfio-user support in DPDK. By doing so, DPDK is enabled > to be an > > alternative I/O device emulation library of building virtualized devices > along with > > high-performance data path in separate processes outside QEMU. It will > be easy for > > hardware vendors to provide virtualized solutions of their hardware > devices by > > implementing emulated device in DPDK. > > > > Except for vfio-user introduced in DPDK, this series also introduces the > first > > emulated device implementation. That is emulated AVF device (avf_emudev) > implemented > > by AVF emulation driver (avf_emudev driver). Emulated AVF device demos > how emulated > > device could be implemented in DPDK. SPDK is also investigating to > implement use case > > for NVMe. > > > > Design overview > > --------------- > > > > +--------------------------------------------------- > ---+ > > | +---------------+ +---------------+ > | > > | | avf_emudev | | avfbe_ethdev | > | > > | | driver | | driver | > | > > | +---------------+ +---------------+ > | > > | | | > | > > | ------------------------------------------- VDEV > BUS | > > | | | > | > > | +---------------+ +--------------+ > | > > +--------------+ | | vdev: | | vdev: | > | > > | +----------+ | | | /path/to/vfio | | avf_emudev_# | > | > > | | Generic | | | +---------------+ +--------------+ > | > > | | vfio-dev | | | | > | > > | +----------+ | | | > | > > | +----------+ | | +----------+ > | > > | | vfio-user| | | | vfio-user| > | > > | | client | |<---|----->| server | > | > > | +----------+ | | +----------+ > | > > | QEMU | | DPDK > | > > +--------------+ +--------------------------------------------------- > ---+ > > > > - vfio-user. Vfio-user in DPDK is referred to the vfio-user protocol > implementation > > playing server role. It provides transport between emulated device and > generic VFIO > > device in QEMU. Emulated device in DPDK and generic VFIO device in QEMU > are working > > together to present VFIO device model to VM. This series introduces > vfio-user > > implementation as a library called librte_vfio_user which is under > lib/librte_vfio_user. > > > > - vdev:/path/to/vfio. It defines the emudev device and binds to vdev bus > driver. The > > emudev device is defined by DPDK applications through command line as '- > -vdev=emu_iavf, > > path=/path/to/socket, id=#' in avf_emudev case. Parameters in command > line include device > > name (emu_iavf) which is used to identify corresponding driver (in this > case, avf_emudev > > driver which implements emudev device of AVF), path=/path/to/socket > which is used to open > > the transport interface to vfio-user client in QEMU, and id which is the > index of emudev > > device. > > > > - avf_emudev driver. It implements emulated AVF device which is the > emudev provider of > > AVF. The avf_emudev_driver offers a few APIs implementation exposed by > emudev device APIs > > for avfbe_ethdev_pmd or application logic to operate. These APIs are > described in > > lib/librte_emudev/rte_emudev.h. > > > > - vdev: avf_emudev_#. The vdev device is defined by DPDK application > through command line > > as '--vdev=net_avfbe,id=#,avf_emu_id=#'.It is associated with emudev > provider of AVF by > > 'avf_emu_id=#'. > > > > - avfbe_ethdev driver. It is a normal ethdev PMD to supply the basic I/O > as backend data > > path of avf_emudev. > > > > Why not rawdev for emulated device > > ---------------------------------- > > Instead of introducing new class emudev, emulated device could be > presented as rawdev. > > However, existing rawdev APIs cannot meet the requirements of emulated > device. There are > > three API categories for emudev. They are emudev device lifecycle > management, backend > > facing APIs, and emudev device provider facing APIs respectively. > Existing rawdev APIs > > could only cover lifecycle management APIs and some of backend facing > APIs. Other APIs, > > even if added to rawdev API are not required by other rawdev > applications. > > > > References > > ---------- > > [1]: https://patchew.org/QEMU/1594913503-52271-1-git-send-email- > thanos.maka...@nutanix.com/ > > [2]: https://wiki.qemu.org/Features/MultiProcessQEMU > > [3]: https://github.com/elmarco/qemu/blob/wip/vfio-user/hw/vfio/libvfio- > user.c > > > > Chenbo Xia (2): > > vfio_user: Add library for vfio over socket > > emudev: Add library for emulated device > > > > lib/librte_emudev/rte_emudev.h | 315 +++++++++++++++++++++++++ > > lib/librte_vfio_user/rte_vfio_user.h | 335 +++++++++++++++++++++++++++ > > 2 files changed, 650 insertions(+) > > create mode 100644 lib/librte_emudev/rte_emudev.h > > create mode 100644 lib/librte_vfio_user/rte_vfio_user.h > > > > This looks good, but it would be good to have an example or way to > integrate > it into a test framework. One of the agree upon principles by the tech > board > is "all new features should have test cases". There have been a lot of > exceptions though.
Thanks a lot for spending time on the review! Since our example is one ethdev PMD driving an emulated vfio-user device, it should be easy for APP like testpmd to test once a vfio-user client is ready (in QEMU or other place). And yes, we will ensure the libs will be into a test framework 😊. Thanks! Chenbo
