On Sun, 12 Apr 2020 09:10:49 -0700 Ravi Kerur <rke...@gmail.com> wrote:
> Hi, > > I use Intel NICs for PF and VF devices. VFs are assigned to virtual > machines and PF is used on the Host. I have intel-iommu=on on GRUB which > enables DMAR and IOMMU capabilities (checked via 'dmesg | grep -e IOMMU -e > DMAR) and use DPDK for datapath acceleration. > > Couple of clarifications I need in terms of vfio-pci driver usage > > (1) with intel-iommu=pt (Passthrough mode), PF device on host can bind to > either igb_uio or vfio-pci driver and similarly VF devices assigned to each > VM can bind to either igb_uio or vfio-pci driver via Qemu Note that the actual option is 'intel_iommu=on iommu=pt'. > (2) with intel-iommu=on (IOMMU enabled), PF device on host must bind to > vfio-pci driver and similarly VF devices assigned to each VM much bind to > vfio-pci driver. When IOMMU is enabled, only vfio-pci should be used? When an IOMMU is present, we refer to the address space through which a device performs DMA as the I/O Virtual Address space, or IOVA. When the IOMMU is in passthrough mode, we effectively create an identity mapping of physical addresses through the IOVA space. Therefore to program a device to perform a DMA to user memory, the user only needs to perform a virtual to physical translation on the address and the device can DMA directly with that physical address thanks to the identity map. When we're not in passthrough mode, we need to actually create a mapping through the IOMMU to allow the device to access that physical memory. VFIO is the only userspace driver interface that I'm aware of that provides this latter functionality. Therefore, yes, if you have the IOMMU enabled and not in passthrough mode, your userspace driver needs support for programming the IOMMU, which vfio-pci provides. Also, having both the PF and VFs owned by userspace drivers presents some additional risks, for example the PF may have access to the data accessed by the VF, or at least be able to deny service to the VF. There have been various hacks around this presented by the DPDK community, essentially enabling SR-IOV underneath vfio-pci, without the driver's knowledge. These are very much dis-recommended, IMO. However, we have added SR-IOV support to the vfio-pci driver in kernel v5.7 and DPDK support is under development, which represents this trust and collaboration between PF and VF drivers using a new VF token concept. I'd encourage you to look for this if your configuration does require both PF and VF drivers in userspace. A much more normal configuration to this point has been that the PF makes use of a host driver (ex. igb, ixgbe, i40e, etc.) while the VF is bound to vfio-pci for userspace drivers. In this configuration the host driver is considered trusted and we don't need to invent new mechanisms to indicate collaboration between userspace drivers. Thanks, Alex _______________________________________________ vfio-users mailing list vfio-users@redhat.com https://www.redhat.com/mailman/listinfo/vfio-users
