On 29-Jan-18 10:35 PM, Ravi Kerur wrote:
Hi Burakov,
When using vfio-pci on host both VF and PF interfaces works fine with
dpdk i.e. I don't see DMAR fault messages anymore. However, when I
attach a VF interface to a VM and start DPDK with vfio-pci inside VM I
still see DMAR fault messages on host. Both host and VM are booted with
'intel-iommu=on' on GRUB. Ping from VM with DPDK/vfio-pci doesn't work
(I think it's expected because of DMAR faults), however, when VF
interface uses ixgbevf driver ping works.
Following are some details
/*****************On VM***************/
dpdk-devbind -s
Network devices using DPDK-compatible driver
============================================
0000:00:07.0 '82599 Ethernet Controller Virtual Function' drv=vfio-pci
unused=ixgbevf
Network devices using kernel driver
===================================
0000:03:00.0 'Device 1041' if=eth0 drv=virtio-pci unused=vfio-pci *Active*
0000:04:00.0 'Device 1041' if=eth1 drv=virtio-pci unused=vfio-pci
0000:05:00.0 'Device 1041' if=eth2 drv=virtio-pci unused=vfio-pci
Other network devices
=====================
<none>
Crypto devices using DPDK-compatible driver
===========================================
<none>
Crypto devices using kernel driver
==================================
<none>
Other crypto devices
====================
<none>
00:07.0 Ethernet controller: Intel Corporation 82599 Ethernet Controller
Virtual Function (rev 01)
Subsystem: Intel Corporation 82599 Ethernet Controller Virtual
Function
Control: I/O- Mem+ BusMaster- SpecCycle- MemWINV- VGASnoop-
ParErr- Stepping- SERR- FastB2B- DisINTx+
Status: Cap+ 66MHz- UDF- FastB2B- ParErr- DEVSEL=fast >TAbort-
<TAbort- <MAbort- >SERR- <PERR- INTx-
Region 0: Memory at fda00000 (64-bit, prefetchable) [size=16K]
Region 3: Memory at fda04000 (64-bit, prefetchable) [size=16K]
Capabilities: [70] MSI-X: Enable+ Count=3 Masked-
Vector table: BAR=3 offset=00000000
PBA: BAR=3 offset=00002000
Capabilities: [a0] Express (v1) Root Complex Integrated
Endpoint, MSI 00
DevCap: MaxPayload 128 bytes, PhantFunc 0
ExtTag- RBE-
DevCtl: Report errors: Correctable- Non-Fatal- Fatal-
Unsupported-
RlxdOrd- ExtTag- PhantFunc- AuxPwr- NoSnoop-
MaxPayload 128 bytes, MaxReadReq 128 bytes
DevSta: CorrErr- UncorrErr- FatalErr- UnsuppReq-
AuxPwr- TransPend-
Capabilities: [100 v1] Advanced Error Reporting
UESta: DLP- SDES- TLP- FCP- CmpltTO- CmpltAbrt-
UnxCmplt- RxOF- MalfTLP- ECRC- UnsupReq- ACSViol-
UEMsk: DLP- SDES- TLP- FCP- CmpltTO- CmpltAbrt-
UnxCmplt- RxOF- MalfTLP- ECRC- UnsupReq- ACSViol-
UESvrt: DLP- SDES- TLP- FCP- CmpltTO- CmpltAbrt-
UnxCmplt- RxOF- MalfTLP- ECRC- UnsupReq- ACSViol-
CESta: RxErr- BadTLP- BadDLLP- Rollover- Timeout-
NonFatalErr-
CEMsk: RxErr- BadTLP- BadDLLP- Rollover- Timeout-
NonFatalErr-
AERCap: First Error Pointer: 00, GenCap- CGenEn-
ChkCap- ChkEn-
Kernel driver in use: vfio-pci
Kernel modules: ixgbevf
/***************on Host*************/
dmesg | grep DMAR
...
[ 978.268143] DMAR: DRHD: handling fault status reg 2
[ 978.268147] DMAR: [DMA Read] *Request device [04:10.0]* fault addr
33a128000 [fault reason 06] PTE Read access is not set
[ 1286.677726] DMAR: DRHD: handling fault status reg 102
[ 1286.677730] DMAR: [DMA Read] Request device [04:10.0] fault addr
fb663000 [fault reason 06] PTE Read access is not set
[ 1676.436145] DMAR: DRHD: handling fault status reg 202
[ 1676.436149] DMAR: [DMA Read] Request device [04:10.0] fault addr
33a128000 [fault reason 06] PTE Read access is not set
[ 1734.433649] DMAR: DRHD: handling fault status reg 302
[ 1734.433652] DMAR: [DMA Read] Request device [04:10.0] fault addr
33a128000 [fault reason 06] PTE Read access is not set
[ 2324.428938] DMAR: DRHD: handling fault status reg 402
[ 2324.428942] DMAR: [DMA Read] Request device [04:10.0] fault addr
7770c000 [fault reason 06] PTE Read access is not set
[ 2388.553640] DMAR: DRHD: handling fault status reg 502
[ 2388.553643] DMAR: [DMA Read] *Request device [04:10.0]* fault addr
33a128000 [fault reason 06] PTE Read access is not set
Going back to this, i would like to suggest run a few tests to ensure
that we have all information that we can gather.
First of all, i'm assuming that you're using native ixgbe Linux driver
on the host, and that you're only passing through the VF device to the
VM using VFIO. Is my understanding correct here?
Now, let's forget about the iommu=pt and igb_uio for a moment. Boot both
your host and your VM with iommu=on and intel_iommu=on (or whatever
command-line enables full IOMMU support on both host and guest) and do
the same tests you've done before. Do you still see your issues?
It would also be very useful to also try native Linux kernel driver on
the guest *with traffic forwarding* and see how it works in your VM.
Therefore i would suggest you to compile DPDK with PCAP support, bind
your (VM) interface to native Linux driver, and use the interface via
our pcap driver (creating a vdev should do the trick - please refer to
PCAP PMD documentation [1]). Simple forwarding test should be enough -
just make sure to pass traffic to and from DPDK in both cases, and that
it doesn't give you any DMAR errors.
We can go from there.
[1] http://dpdk.org/doc/guides/nics/pcap_ring.html
--
Thanks,
Anatoly
