On 12/21/20 10:52 AM, Maxime Coquelin wrote:
> Hi Chenbo,
>
> On 12/19/20 7:11 AM, Xia, Chenbo wrote:
>> Hi David,
>>
>>> -----Original Message-----
>>> From: David Marchand <david.march...@redhat.com>
>>> Sent: Friday, December 18, 2020 5:54 PM
>>> To: Xia, Chenbo <chenbo....@intel.com>
>>> Cc: dev <dev@dpdk.org>; Thomas Monjalon <tho...@monjalon.net>; Stephen
>>> Hemminger <step...@networkplumber.org>; Liang, Cunming
>>> <cunming.li...@intel.com>; Lu, Xiuchun <xiuchun...@intel.com>; Li, Miao
>>> <miao...@intel.com>; Wu, Jingjing <jingjing...@intel.com>
>>> Subject: Re: [PATCH 0/8] Introduce emudev library and iavf emudev driver
>>>
>>> On Fri, Dec 18, 2020 at 9:02 AM Chenbo Xia <chenbo....@intel.com> wrote:
>>>>
>>>> This series introduces a new device abstraction called emudev for
>>> emulated
>>>> devices. A new library (librte_emudev) is implemented. The first emudev
>>>> driver is also introduced, which emulates Intel Adaptive Virtual
>>> Function
>>>> (iavf) as a software network device.
>>>>
>>>> This series has a dependency on librte_vfio_user patch series:
>>>> http://patchwork.dpdk.org/cover/85389/
>>>>
>>>> Background & Motivation
>>>> -----------------------
>>>> The disaggregated/multi-process QEMU is using VFIO-over-socket/vfio-user
>>>> as the main transport mechanism to disaggregate IO services from QEMU.
>>>> Therefore, librte_vfio_user is introduced in DPDK to accommodate
>>>> emulated devices for high performance I/O. Although vfio-user library
>>>> provides possibility of emulating devices in DPDK, DPDK does not have
>>>> a device abstraction for emulated devices. A good device abstraction
>>> will
>>>> be useful for applications or high performance data path driver. With
>>>> this consideration, emudev library is designed and implemented. It also
>>>> make it possbile to keep modular design on emulated devices by
>>> implementing
>>>> data path related logic in a standalone driver (e.g., an ethdev driver)
>>>> and keeps the unrelated logic in the emudev driver.
>>>
>>> Since you mention performance, how does it compare to vhost-user/virtio?
>>
>> I think it depends on the device specification (i.e., how complex its data
>> path
>> handling is). A first try on iavf spec shows better performance than virtio
>> in our simple tests.
>
> That's interesting! How big is the performance difference? And how do
> we explain it?
>
> If there are improvements that could be done in the Virtio
> specification, it would be great to know and work on their
> implementations. It worries me a bit that every one is coming with
> his new device emulation every release, making things like live-
> migration difficult to achieve in the future.
I did a quick review of the IAVF emudev driver to understand what other
factors than ring layout could explain a performance gain.
My understanding is that part of the performance gain may come from
following things that are supported/implemented in Vhost-user backend
and not in IAVF driver:
1. Memory hotplug. It seems the datapath is not safe against memory
hotplug in the VM, which causes the memory tables to be updated
asynchronously from the datapath. In order to support it in Vhost-user
library, we had to introduce locks to ensure the datapath isn't
accessing the shared memory while it is being remapped.
2. Live-migration. This feature does not seem supported in the driver,
as I could not find dirty pages tracking mechanism. On Vhost-user side,
supporting implies adding more branch conditions in the hot path, even
when it is not used.
3. No IOMMU support. Same here, this is supported in Vhost-user library,
and adding its support in IAVF driver would introduce some more branches
in the hot path even when not used.
4. Out of bound memory accesses checks. While
rte_iavf_emu_get_dma_vaddr() provides a way to ensure the full requested
length is mapped, the data path does not use it. It does not even ensure
the translated address is non-NULL. It makes it trivial for a malicious
guest to make the hypervisor's vSwitch to crash by simply passing random
values as buffer's address and length. Fixing it is trivial, but it will
add several more checks and loops (if a buffer is spanned across two
pages) in the hot path.
Other than that, there is for sure a performance gain due to all the
features Virtio-net supports that we have to check and handle in the
hotpath, like indirect descriptors or mergeable buffers for example.
Best regards,
Maxime
> Regards,
> Maxime
>
>> Thanks!
>> Chenbo
>>
>>>
>>>
>>> --
>>> David Marchand
>>
