On Fri, Feb 2, 2018 at 12:21 PM, Ravi Kerur <rke...@gmail.com
<mailto:rke...@gmail.com>> wrote:
On Fri, Feb 2, 2018 at 2:28 AM, Burakov, Anatoly
<anatoly.bura...@intel.com <mailto:anatoly.bura...@intel.com>> wrote:
On 01-Feb-18 7:26 PM, Ravi Kerur wrote:
On Thu, Feb 1, 2018 at 2:10 AM, Burakov, Anatoly
<anatoly.bura...@intel.com
<mailto:anatoly.bura...@intel.com>
<mailto:anatoly.bura...@intel.com
<mailto:anatoly.bura...@intel.com>>> wrote:
On 31-Jan-18 9:51 PM, Ravi Kerur wrote:
Hi Anatoly,
Thanks. I am following wiki link below which uses
vIOMMU with
DPDK as a use-case and instantiate VM as specified
with Q35
chipset in Qemu.
https://wiki.qemu.org/Features/VT-d
<https://wiki.qemu.org/Features/VT-d>
<https://wiki.qemu.org/Features/VT-d
<https://wiki.qemu.org/Features/VT-d>>
Qemu-version is 2.11
Host kernel 4.9
Guest kernel 4.4
I can only guess that guest kernel needs an upgrade
in my setup
to work correctly, if versions on my setup rings a
bell on not
having support kindly let me know.
When 'modprobe vfio enable_unsafe_noiommu_node=Y'
is executed on
guest I get following error
...
vfio: unknown parameter
'enable_unsafe_noiommu_node' ignored
...
in guest.
Thanks.
AFAIK kernel 4.4 should have noiommu mode - it was
introduced in
3.1x days. However, in order for that to work, kernel
also has to be
built with this mode enabled. My guess is, whoever is
the supplier
of your kernel, did not do that. You should
double-check the kernel
configuration of your distribution.
However, if you have vIOMMU in QEMU, you shouldn't need
noiommu mode
- "regular" vfio should work fine. noiommu mode should
only be
needed if you know you don't have IOMMU enabled in your
kernel, and
even if you can't enable it, you can still use igb_uio.
Hi Anatoly,
Do you suggest I take this discussion to kvm/qemu mailing
list as I am not sure which component has the issue? I check
dmesg for BIOS physical memory map and address reported as
fault by DMAR is reported by BIOS as usable on both host and vm.
[ 4539.597737] DMAR: [DMA Read] Request device [04:10.0]
fault addr *33a128000 *[fault reason 06] PTE Read access is
not set
dmesg | grep BIOS
[ 0.000000] e820: BIOS-provided physical RAM map:
[ 0.000000] BIOS-e820: [mem
0x0000000000000000-0x000000000009afff] usable
[ 0.000000] BIOS-e820: [mem
0x000000000009b000-0x000000000009ffff] reserved
[ 0.000000] BIOS-e820: [mem
0x00000000000e0000-0x00000000000fffff] reserved
[ 0.000000] BIOS-e820: [mem
0x0000000000100000-0x000000007938afff] usable
[ 0.000000] BIOS-e820: [mem
0x000000007938b000-0x000000007994bfff] reserved
[ 0.000000] BIOS-e820: [mem
0x000000007994c000-0x000000007999cfff] ACPI data
[ 0.000000] BIOS-e820: [mem
0x000000007999d000-0x0000000079f7dfff] ACPI NVS
[ 0.000000] BIOS-e820: [mem
0x0000000079f7e000-0x000000007bd37fff] reserved
[ 0.000000] BIOS-e820: [mem
0x000000007bd38000-0x000000007bd38fff] usable
[ 0.000000] BIOS-e820: [mem
0x000000007bd39000-0x000000007bdbefff] reserved
[ 0.000000] BIOS-e820: [mem
0x000000007bdbf000-0x000000007bffffff] usable
[ 0.000000] BIOS-e820: [mem
0x000000007c000000-0x000000008fffffff] reserved
[ 0.000000] BIOS-e820: [mem
0x00000000fed1c000-0x00000000fed44fff] reserved
[ 0.000000] BIOS-e820: [mem
0x00000000ff000000-0x00000000ffffffff] reserved
[* 0.000000] BIOS-e820: [mem
0x0000000100000000-0x000000407fffffff] usable*
*
*
Kindly let me know your inputs.
Thanks.
-- Thanks,
Anatoly
The "PTE Read not set" error usually indicates that you are
trying to use a non-IOMMU method when you have IOMMU enabled
(i.e. trying to use igb_uio when IOMMU is on). That, to me,
indicates that you do have IOMMU emulation enabled.
I would go about it this way.
First, i'd ensure that your VM has IOMMU emulation enabled and
working. You have mentioned that your QEMU version should have
IOMMU emulation, so let's assume that's the case.
I am not sure of the exact command-line needed to activate the
vIOMMU emulation, but assuming your VM emulates an Intel
processor, your kernel command-line should have "iommu=on
intel_iommu=on" in it. Check /etc/default/grub for
GRUB_CMDLINE_LINUX_DEFAULT value, and if the above values are
not in there, add the above changes, do "update-grub" and reboot
your VM.
If it already did have the necessary kernel configuration, do
"dmesg | grep IOMMU" and look for "IOMMU Enabled". That should
tell you that IOMMU is enabled and working in the kernel.
After that, you can modprobe vfio and vfio-pci, bind NICs to it,
and it should be working. Please bear in mind that all of that
is how i would've gone about it if i had similar problems on
baremetal, but i'm hoping all of it is applicable to VM's. So,
either disable IOMMU and use igb_uio, or enable IOMMU and use
VFIO. Both will work.
Thanks for your help so far on this. I have everything enabled as
you have mentioned in your email. Below is the error from EAL when
DPDK is run on a VM. I am referring EAL code to check why it is failing.
************VM DPDK errors***********
EAL: Detected 14 lcore(s)
EAL: Probing VFIO support...
EAL: VFIO support initialized
EAL: WARNING: cpu flags constant_tsc=yes nonstop_tsc=no -> using
unreliable clock cycles !
EAL: PCI device 0000:00:07.0 on NUMA socket -1
EAL: Invalid NUMA socket, default to 0
EAL: probe driver: 8086:10ed net_ixgbe_vf
EAL: using IOMMU type 1 (Type 1)
*EAL: RTE_IOVA_VA 140051367329792
EAL: cannot set up DMA remapping, error 14 (Bad address)
EAL: 0000:00:07.0 DMA remapping failed, error 14 (Bad address)*
EAL: Requested device 0000:00:07.0 cannot be used
EAL: PCI device 0000:03:00.0 on NUMA socket -1
EAL: Invalid NUMA socket, default to 0
EAL: probe driver: 1af4:1041 net_virtio
EAL: PCI device 0000:04:00.0 on NUMA socket -1
EAL: Invalid NUMA socket, default to 0
EAL: probe driver: 1af4:1041 net_virtio
EAL: PCI device 0000:05:00.0 on NUMA socket -1
EAL: Invalid NUMA socket, default to 0
EAL: probe driver: 1af4:1041 net_virtio
EAL: No probed ethernet devices
Interactive-mode selected
USER1: create a new mbuf pool <mbuf_pool_socket_0>: n=171456,
size=2176, socket=0
********************VM************
root@sriov-ravi-01:~# lsmod | grep vfio
*vfio_pci 40960 1
vfio_virqfd 16384 1 vfio_pci
vfio_iommu_type1 20480 1
vfio 28672 5 vfio_iommu_type1,vfio_pci
irqbypass 16384 3 kvm,vfio_pci*
root@sriov-ravi-01:~# dmesg | grep -e DMAR -e IOMMU
[ 0.000000] ACPI: DMAR 0x000000007FFE209D 000050 (v01 BOCHS
BXPCDMAR 00000001 BXPC 00000001)
*[ 0.000000] DMAR: IOMMU enabled*
[ 9.423839] DMAR: Host address width 39
[ 9.424640] DMAR: DRHD base: 0x000000fed90000 flags: 0x1
[ 9.425754] DMAR: dmar0: reg_base_addr fed90000 ver 1:0 cap
12008c22260286 ecap f00f5e
[ 9.427314] DMAR: ATSR flags: 0x1
[ 9.428047] DMAR-IR: IOAPIC id 0 under DRHD base 0xfed90000 IOMMU 0
[ 9.429255] DMAR-IR: Queued invalidation will be enabled to
support x2apic and Intr-remapping.
[ 9.433096] DMAR-IR: Enabled IRQ remapping in x2apic mode
[ 11.167334] DMAR: No RMRR found
[ 11.168855] DMAR: dmar0: Using Queued invalidation
[ 12.119060] DMAR: Setting RMRR:
[ 12.123734] DMAR: Prepare 0-16MiB unity mapping for LPC
[ 12.124787] DMAR: Setting identity map for device 0000:00:1f.0
[0x0 - 0xffffff]
[ 12.392826] DMAR: Intel(R) Virtualization Technology for Directed I/O
root@sriov-ravi-01:~# dpdk-devbind -s
Network devices using DPDK-compatible driver
============================================
*0000:00:07.0 '82599 Ethernet Controller Virtual Function 10ed'
drv=vfio-pci unused=*
Network devices using kernel driver
===================================
0000:03:00.0 'Device 1041' if=eth0 drv=virtio-pci unused=vfio-pci
*Active*
0000:04:00.0 'Device 1041' if=eth1 drv=virtio-pci unused=vfio-pci
0000:05:00.0 'Device 1041' if=eth2 drv=virtio-pci unused=vfio-pci
Other Network devices
=====================
<none>
Crypto devices using DPDK-compatible driver
===========================================
<none>
Crypto devices using kernel driver
==================================
<none>
Other Crypto devices
====================
<none>
Eventdev devices using DPDK-compatible driver
=============================================
<none>
Eventdev devices using kernel driver
====================================
<none>
Other Eventdev devices
======================
<none>
Mempool devices using DPDK-compatible driver
============================================
<none>
Mempool devices using kernel driver
===================================
<none>
Other Mempool devices
=====================
<none>
root@sriov-ravi-01:~#
root@sriov-ravi-01:/home/deployuser/dpdk-17.11/usertools# dmesg |
grep -i vfio
[ 77.676711] VFIO - User Level meta-driver version: 0.3
[ 149.013806] VFIO - User Level meta-driver version: 0.3
[ 275.876440] VFIO - User Level meta-driver version: 0.3
Thanks,
Ravi
Additonal information. On VM following message is seen
[ 298.714974] DMAR: intel_iommu_map: iommu width (39) is not sufficient
for the mapped address (7f7300000000)
[ 1494.673406] DMAR: intel_iommu_map: iommu width (39) is not sufficient
for the mapped address (7fa580000000)
ON VM, DMAR reports address width = 39
root@sriov-ravi-01:/home/deployuser/dpdk-17.11/usertools# dmesg | grep
-e IOMMU -e DMAR
[ 0.000000] ACPI: DMAR 0x000000007FFE209D 000050 (v01 BOCHS BXPCDMAR
00000001 BXPC 00000001)
[ 0.000000] DMAR: IOMMU enabled
[ 9.209145] DMAR: Host address width 39
On Host, DMAR reports address width = 46
# dmesg | grep -e DMAR -e IOMMU
[ 0.000000] ACPI: DMAR 0x000000007999BAD0 0000E0 (v01 ALASKA A M I
00000001 INTL 20091013)
[ 0.000000] DMAR: IOMMU enabled
[ 0.519368] DMAR: Host address width 46
[ 0.527243] DMAR: DRHD base: 0x000000fbffc000 flags: 0x0
Searching for IOMMU errors found following link from RedHat which points
to DPDK 17.11 issue.
https://bugzilla.redhat.com/show_bug.cgi?id=1530957
Thanks.
--
Thanks,
Anatoly
