Hi Joshua, We'd need the whole dmesg. It looks like vfio-pci either failed to load or is not usable (probably an IOMMU issue) but it might be something else. Alternatively, you can try to disable vfio IOMMU support and bind to vfio again: ~# echo 1 | sudo tee /sys/module/vfio/parameters/enable_unsafe_noiommu_mode ~# sudo dpdk-devbind --bind=vfio-pci 0000:13:00.0
Best ben > -----Original Message----- > From: vpp-dev@lists.fd.io <vpp-dev@lists.fd.io> On Behalf Of Joshua Moore > Sent: mardi 29 septembre 2020 00:26 > To: Joshua Moore <j...@jcm.me> > Cc: Damjan Marion <dmar...@me.com>; Benoit Ganne (bganne) > <bga...@cisco.com>; vpp-dev@lists.fd.io > Subject: Re: [vpp-dev] VPP on ESXI with i40evf (SR-IOV Passthrough) Driver > > Sorry, previous dmesg was greped on my VMXNET3 adapter, not the i40evf. > Correct dmesg: > > jmoore@test:~$ dmesg | grep 0000:13:00.0 > [ 0.259249] pci 0000:13:00.0: [8086:154c] type 00 class 0x020000 > [ 0.261432] pci 0000:13:00.0: reg 0x10: [mem 0xe7af0000-0xe7afffff > 64bit pref] > [ 0.266767] pci 0000:13:00.0: reg 0x1c: [mem 0xe7aec000-0xe7aeffff > 64bit pref] > [ 0.272845] pci 0000:13:00.0: disabling ASPM on pre-1.1 PCIe device. > You can enable it with 'pcie_aspm=force' > [ 1.179790] iommu: Adding device 0000:13:00.0 to group 8 > [ 2.196529] i40evf 0000:13:00.0: Multiqueue Enabled: Queue pair count = > 4 > [ 2.196799] i40evf 0000:13:00.0: MAC address: 00:0c:29:58:7f:b5 > [ 2.196865] i40evf 0000:13:00.0: GRO is enabled > [ 2.510262] i40evf 0000:13:00.0 ens224: renamed from eth2 > > > On Mon, Sep 28, 2020 at 4:57 PM Joshua Moore via lists.fd.io > <http://lists.fd.io> <j=jcm...@lists.fd.io <mailto:jcm...@lists.fd.io> > > wrote: > > > Hi Damjan, > > I am running Ubuntu 18.04 LTS with kernel 4.15.0-118-generic > > > See below dmesg output. > > jmoore@test:~$ dmesg | grep 0000:03:00.0 > [ 0.223459] pci 0000:03:00.0: [15ad:07b0] type 00 class 0x020000 > [ 0.225126] pci 0000:03:00.0: reg 0x10: [mem 0xfd4fc000- > 0xfd4fcfff] > [ 0.227304] pci 0000:03:00.0: reg 0x14: [mem 0xfd4fd000- > 0xfd4fdfff] > [ 0.229121] pci 0000:03:00.0: reg 0x18: [mem 0xfd4fe000- > 0xfd4fffff] > [ 0.231298] pci 0000:03:00.0: reg 0x1c: [io 0x4000-0x400f] > [ 0.237119] pci 0000:03:00.0: reg 0x30: [mem 0x00000000- > 0x0000ffff pref] > [ 0.237550] pci 0000:03:00.0: supports D1 D2 > [ 0.237551] pci 0000:03:00.0: PME# supported from D0 D1 D2 D3hot > D3cold > [ 0.237774] pci 0000:03:00.0: disabling ASPM on pre-1.1 PCIe > device. You can enable it with 'pcie_aspm=force' > [ 0.353290] pci 0000:03:00.0: BAR 6: assigned [mem 0xfd400000- > 0xfd40ffff pref] > [ 1.179463] iommu: Adding device 0000:03:00.0 to group 6 > [ 2.108455] vmxnet3 0000:03:00.0: # of Tx queues : 8, # of Rx > queues : 8 > [ 2.110321] vmxnet3 0000:03:00.0 eth0: NIC Link is Up 10000 Mbps > [ 2.471328] vmxnet3 0000:03:00.0 ens160: renamed from eth0 > > > > > On Mon, Sep 28, 2020 at 1:02 PM Damjan Marion <dmar...@me.com > <mailto:dmar...@me.com> > wrote: > > > > What message do you see in dmesg? What is the kernel version? > > > > On 28.09.2020., at 19:47, Joshua Moore <j...@jcm.me > <mailto:j...@jcm.me> > wrote: > > Sorry, I'm still hitting an issue where I cannot create > the interface in VPP: > > > vpp# create interface avf 0000:13:00.0 > create interface avf: device not bound to 'vfio- > pci' or 'uio_pci_generic' kernel module > > > > So I tried to bind the NIC to vfio-pci: > > > root@test:~# modprobe vfio-pci > root@test:~# /usr/local/vpp/vpp- > config/scripts/dpdk-devbind.py -s > > Network devices using DPDK-compatible driver > ============================================ > <none> > > Network devices using kernel driver > =================================== > 0000:13:00.0 'Ethernet Virtual Function 700 > Series' if=ens224 drv=i40evf unused= > > root@test:~# /usr/local/vpp/vpp- > config/scripts/dpdk-devbind.py --bind vfio-pci 13:00.0 > Error - no supported modules(DPDK driver) are > loaded > > > > Thoughts? > > On Mon, Sep 28, 2020 at 11:43 AM Benoit Ganne (bganne) > <bga...@cisco.com <mailto:bga...@cisco.com> > wrote: > > > Hi Johsua, > > Your understanding is correct, however you do > not > need to setup the VFs if it is already correctly setup by ESXI. > Just create the AVF interface directly by > specifying the VF PCI address. > > ben > > > -----Original Message----- > > From: Joshua Moore <j...@jcm.me > <mailto:j...@jcm.me> > > > Sent: lundi 28 septembre 2020 17:48 > > To: Benoit Ganne (bganne) <bga...@cisco.com > <mailto:bga...@cisco.com> > > > Cc: vpp-dev@lists.fd.io <mailto:vpp- > d...@lists.fd.io> > > Subject: Re: [vpp-dev] VPP on ESXI with i40evf > (SR-IOV Passthrough) Driver > > > > Hello Benoit, > > > > Looking at the script for AVF, it states: > > > > # Setup one VF on PF 0000:3b:00.0 and assign > MAC > address > > setup 0000:3b:00.0 00:11:22:33:44:00 > > # Setup one VF on PF 0000:3b:00.1 and assign > MAC > address > > setup 0000:3b:00.1 00:11:22:33:44:01 > > > > This seems to assume the entire PF NIC is > exposed to the VM and the VM is > > responsible for owning the configuration of > the > WHOLE PF to setup the VF. > > This also makes sense to me considering that > the > script is looking for > > i40en driver (physical) and not i40evf driver > (virtual). My understanding > > is that this will not work with my ESXI setup > as > ESXI owns the > > configuration of the PF (physical NIC) and is > assigning the VFs from the > > NIC end is exposing just the VF to the VM. > > > > Does this make sense or am I misunderstanding > something? > > > > If so, then how can the AVF plugin/driver > consume just the VF NIC already > > assigned to the VM and not try to setup a new > VF? > > > > > > Thanks! > > > > -Josh > > > > On Mon, Sep 28, 2020 at 2:40 AM Benoit Ganne > (bganne) <bga...@cisco.com <mailto:bga...@cisco.com> > > <mailto:bga...@cisco.com > <mailto:bga...@cisco.com> > > wrote: > > > > > > Hi, > > > > It should work with AVF as it is using > VFs, not PF, see > > > https://docs.fd.io/vpp/21.01/d1/def/avf_plugin_doc.html > > You should bind the VF with vfio-pci 1st > though, so that it is > > usable by userspace drivers such as VPP AVF > plugin. > > If your system crashes when doing so it > is > a bug with your system. > > > > Best > > ben > > > > > -----Original Message----- > > > From: vpp-dev@lists.fd.io <mailto:vpp- > d...@lists.fd.io> <mailto:vpp-dev@lists.fd.io <mailto:vpp-dev@lists.fd.io> > > <vpp- > > d...@lists.fd.io <mailto:d...@lists.fd.io> > <mailto:vpp-dev@lists.fd.io <mailto:vpp-dev@lists.fd.io> > > On Behalf Of > j...@jcm.me <mailto:j...@jcm.me> > > <mailto:j...@jcm.me <mailto:j...@jcm.me> > > > > Sent: lundi 28 septembre 2020 01:29 > > > To: vpp-dev@lists.fd.io <mailto:vpp- > d...@lists.fd.io> <mailto:vpp-dev@lists.fd.io <mailto:vpp-dev@lists.fd.io> > > > > > Subject: [vpp-dev] VPP on ESXI with > i40evf (SR-IOV Passthrough) > > Driver > > > > > > Hello, > > > > > > Is there any support for VPP to talk > directly to ESXI-assigned VFs > > via SR- > > > IOV? I saw the AVF plugin but I don't > want VPP to control the > > whole PF > > > (physical NIC) but rather would like > to > have ESXI control the > > mapping of > > > VFs (SR-IOV) and VPP (or DPDK) consume > the VF natively in the VM > > so that I > > > can run multiple VMs on the same > physical NIC while benefiting > > from > > > bypassing the vSwitch in ESXI. Right > now > I'm running VPP on a > > Ubuntu 18.04 > > > VM and I see the SR-IOV NIC as an > i40evf > driver. > > > > > > I tried binding the SR-IOV NIC to the > vfio driver but this causes > > the CPU > > > of the VM to skyrocket and crash. I > don't think using vfio is the > > right > > > approach and feel like the solution > here > is really simple. Any > > > suggestions? > > > > > > > > > > >
-=-=-=-=-=-=-=-=-=-=-=- Links: You receive all messages sent to this group. View/Reply Online (#17572): https://lists.fd.io/g/vpp-dev/message/17572 Mute This Topic: https://lists.fd.io/mt/77164974/21656 Group Owner: vpp-dev+ow...@lists.fd.io Unsubscribe: https://lists.fd.io/g/vpp-dev/unsub [arch...@mail-archive.com] -=-=-=-=-=-=-=-=-=-=-=-
