If I remember correctly, you mentioned in early email that you do _NOT_ modify
the startup.conf file for VPP. Can you confirm that?
By looking at the startup.conf file coming with VPP v18.04 the default mechanism
driver used is vfio-pci. That as mentioned before (and found out by you too)
won't work.Also, by default the DPDK section is completely commented out so all
the defaults are being used...
Can you please take a look at your startup.conf file and make sure to use
igb_uio by specifying "uio-driver igb_uio"
And, first of all, make sure the dpdk section is not commented out?
I think something like the below should help you to get going...
dpdk { uio-driver igb_uio }
- Marco
On Wed, 2018-04-11 at 16:12 +0900, Moon-Sang Lee wrote:
> I'm back with a little progress.
> Anyway, I found that vpp interface has MTU size as 9220B.
> And I just tried to change it to 1500B as below.
> # vppctl set interface mtu 1500 VirtualFunctionEthernet0/8/0
>
> It works!
> My VM with VPP/DPDK ping/pong to outside server via VF of Intel 82599 10G NIC.
> But, I could not understand why MTU configuration affect to simple ping test.
>
> In addition, I need to measure this VM's performance with pktgen-dpdk which
> has no MTU option as far as I know.
> When I shoot packets from pktgen, VM does not receive packets nor ping does
> not work again.
> (i.e. pktgen and VM are directly connected with 10G link.)
>
> I appreciate any comment.
>
>
>
> On Tue, Apr 10, 2018 at 9:16 PM, Moon-Sang Lee <sang0...@gmail.com> wrote:
> > On Tue, Apr 10, 2018 at 8:24 PM, Marco Varlese <mvarl...@suse.de> wrote:
> > > On Tue, 2018-04-10 at 19:33 +0900, Moon-Sang Lee wrote:
> > > > Thanks for your interest, Marco.
> > > > I follows the intel guideline, "As an SR-IOV VF network adapter using a
> > > > KVM virtual network pool of adapters"
> > > > from https://software.intel.com/en-us/articles/configure-sr-iov-network-
> > > > virtual-functions-in-linux-kvm.
> > > >
> > > > In summary, I modprobe ixgbe on host side and creates one VF per PF.
> > > > When I start VM using virsh, libvirt binds the VF to vfio-pci in host
> > > > side.
> > > > After VM finishes booting, I login to VM and bind the VF to igb_uio
> > > > using dpdk-dev command.
> > > > (i.e. only igb_uio works and other drivers like uio_pci_generic and
> > > > vfio-pci fail to bind VF in VM side.)
> > > Yes, that's expected.
> > > If you want to use vfio-pci in the VM you'll need to enable the "no-
> > > iommu":
> > > # echo 1 > /sys/module/vfio/parameters/enable_unsafe_noiommu_mode
> >
> > well, I cannot find such file in host nor VM.
> >
> > > > I don't edit startup.conf in VM, but I just bind the VF to dpdk-
> > > > compatible driver, igb_uio, inside VM.
> > > > With above configuration, I can bind the VF to guest kernel driver,
> > > > ixgbevf, and also to DPDK PMD, igb_uio.
> > > > As a result, I can run VPP without DPDK using ixgbevf, and also DPDK
> > > > applications using igb_uio.
> > > > (i.e. I successfully runs l2fwd/l3fwd sample applications inside VM, so
> > > > I guess VF biding has no problem.)
> > > >
> > > > However, I cannot run VPP with DPDK and I suspect hugepage is related to
> > > > this problem as shown in my VPP log.
> > > So, what does the command "cat /proc/meminfo |grep HugePages_" shows?
> >
> > well, 'cat /proc/meminfo |grep HugePages' seems ok even though I cannot find
> > any rtemap files neither in /dev/hugepages nor /run/vpp/hugepages.
> > When I run dpdk application, I can see rtemap files in /dev/hugepages.
> >
> > HugePages_Total: 1024
> > HugePages_Free: 972
> > HugePages_Rsvd: 0
> > HugePages_Surp: 0
> > Hugepagesize: 2048 kB
> >
> > FYI.Apr 10 05:12:35 ubuntu1604 /usr/bin/vpp[1720]: dpdk_config:1271: EAL
> > init args: -c 1 -n 4 --huge-dir /run/vpp/hugepages --file-prefix vpp -b
> > 0000:00:05.0 -b 0000:00:06.0 -b 0000:00:07.0 --master-lcore 0 --socket-mem
> > 64
> >
> >
> > > > Every packet sent from the VM is traversed via VF to the opposite side,
> > > > a pktgen server.
> > > > And pktgen replies to those packets, but VM does not receive those
> > > > replies.
> > > > (i.e. I ping to pktgen port from VM, where host server port is directly
> > > > linked to pktgen server port.)
> > > >
> > > > Here is my vpp-config script.
> > > > I test ping after running this script.
> > > >
> > > > #!/bin/sh
> > > >
> > > > vppctl enable tap-inject
> > > > vppctl create loopback interface
> > > > vppctl set interface state loop0 up
> > > >
> > > > vppctl show interface
> > > > vppctl show hardware
> > > > vppctl set interface state VirtualFunctionEthernet0/6/0 up
> > > > vppctl set interface state VirtualFunctionEthernet0/7/0 up
> > > > vppctl set interface ip address loop0 2.2.2.2/32
> > > > vppctl set interface ip address VirtualFunctionEthernet0/6/0
> > > > 192.168.0.1/24
> > > > vppctl set interface ip address VirtualFunctionEthernet0/7/0
> > > > 192.168.1.1/24
> > > > vppctl show interface address
> > > > vppctl show tap-inject
> > > >
> > > > ip addr add 2.2.2.2/32 dev vpp0
> > > > ip addr add 192.168.0.1/24 dev vpp1
> > > > ip addr add 192.168.1.1/24 dev vpp2
> > > > ip link set dev vpp0 up
> > > > ip link set dev vpp1 up
> > > > ip link set dev vpp2 up
> > > >
> > > >
> > > >
> > > > On Tue, Apr 10, 2018 at 4:51 PM, Marco Varlese <mvarl...@suse.de> wrote:
> > > > > On Mon, 2018-04-09 at 22:53 +0900, Moon-Sang Lee wrote:
> > > > > > I've configured a VM with KVM, and the VM is intended to run VPP
> > > > > > with DPDK.
> > > > > > In particular, the VM is connected to one of VFs. (i.e. SR-IOV)
> > > > > > I can run DPDK sample applications,including l2fwd and l3fwd, in the
> > > > > > VM,
> > > > > > therefore I guess VM is successfully connected to the outside-
> > > > > > world(pktgen server) via VFs.
> > > > > >
> > > > > > However, I cannot receive a packet when I run VPP/DPDK.
> > > > > > I can see TX packets from the VM on the opposite side, pktgen
> > > > > > server,
> > > > > > but the VM does not receive any reply from pktgen server which
> > > > > > reports RX/TX packet count.
> > > > > > (i.e. arping/ping from VM arrives in pktgen, but the reply from
> > > > > > pktgen is not received in VM.)
> > > > > > I found some strange log messages regarding vpp launching as below.
> > > > > >
> > > > > > I appreciate for any comment.
> > > > > > Thanks in advance...
> > > > > >
> > > > > > - Host NIC: Intel 82599 10G NIC (i.e. VF binding with vfio-pci)
> > > > > > - VM: 1 socket 4 vCPU
> > > > > > - VPP: 18.04
> > > > > > - DPDK binding: igb_uio
> > > > > It isn't clear to me who manages the PF in the host and how you
> > > > > created the VFs (kernel module or via DPDK binding)?
> > > > >
> > > > > Second, what do you mean by DPDK binding in the last line above?
> > > > > Is that what you have configured in startup.conf in the VM for VPP to
> > > > > be used?
> > > > >
> > > > > If so, what is the different between VF binding and DPDK binding in
> > > > > your short setup summary above? I'm confused by reading vfio-pci in
> > > > > one place and then igb_uio later on.
> > > > >
> > > > > Can you provide us with the startup.conf you have in the VM?
> > > > >
> > > > > Finally, if you are interested in using vfio-pci then you'll need to
> > > > > have the no-IOMMU enabled otherwise you can't use VFIO in the VM...
> > > > > probably, the easiest would be to use igb_uio everywhere...
> > > > >
> > > > > > root@xenial-vpp-frr:~# vpp -c /etc/vpp/startup.conf
> > > > > > vlib_plugin_early_init:359: plugin path /usr/lib/vpp_plugins
> > > > > > load_one_plugin:187: Loaded plugin: acl_plugin.so (Access Control
> > > > > > Lists)
> > > > > > load_one_plugin:187: Loaded plugin: avf_plugin.so (Intel Adaptive
> > > > > > Virtual Function (AVF) Device Plugin)
> > > > > > load_one_plugin:189: Loaded plugin: cdp_plugin.so
> > > > > > load_one_plugin:187: Loaded plugin: dpdk_plugin.so (Data Plane
> > > > > > Development Kit (DPDK))
> > > > > > load_one_plugin:187: Loaded plugin: flowprobe_plugin.so (Flow per
> > > > > > Packet)
> > > > > > load_one_plugin:187: Loaded plugin: gbp_plugin.so (Group Based
> > > > > > Policy)
> > > > > > load_one_plugin:187: Loaded plugin: gtpu_plugin.so (GTPv1-U)
> > > > > > load_one_plugin:187: Loaded plugin: igmp_plugin.so (IGMP messaging)
> > > > > > load_one_plugin:187: Loaded plugin: ila_plugin.so (Identifier-
> > > > > > locator addressing for IPv6)
> > > > > > load_one_plugin:187: Loaded plugin: ioam_plugin.so (Inbound OAM)
> > > > > > load_one_plugin:117: Plugin disabled (default): ixge_plugin.so
> > > > > > load_one_plugin:187: Loaded plugin: kubeproxy_plugin.so (kube-proxy
> > > > > > data plane)
> > > > > > load_one_plugin:187: Loaded plugin: l2e_plugin.so (L2 Emulation)
> > > > > > load_one_plugin:187: Loaded plugin: lacp_plugin.so (Link Aggregation
> > > > > > Control Protocol)
> > > > > > load_one_plugin:187: Loaded plugin: lb_plugin.so (Load Balancer)
> > > > > > load_one_plugin:187: Loaded plugin: memif_plugin.so (Packet Memory
> > > > > > Interface (experimetal))
> > > > > > load_one_plugin:187: Loaded plugin: nat_plugin.so (Network Address
> > > > > > Translation)
> > > > > > load_one_plugin:187: Loaded plugin: pppoe_plugin.so (PPPoE)
> > > > > > load_one_plugin:187: Loaded plugin: router.so (router)
> > > > > > load_one_plugin:187: Loaded plugin: srv6ad_plugin.so (Dynamic SRv6
> > > > > > proxy)
> > > > > > load_one_plugin:187: Loaded plugin: srv6am_plugin.so (Masquerading
> > > > > > SRv6 proxy)
> > > > > > load_one_plugin:187: Loaded plugin: srv6as_plugin.so (Static SRv6
> > > > > > proxy)
> > > > > > load_one_plugin:187: Loaded plugin: stn_plugin.so (VPP Steals the
> > > > > > NIC for Container integration)
> > > > > > load_one_plugin:187: Loaded plugin: tlsmbedtls_plugin.so (mbedtls
> > > > > > based TLS Engine)
> > > > > > load_one_plugin:187: Loaded plugin: tlsopenssl_plugin.so (openssl
> > > > > > based TLS Engine)
> > > > > > load_one_plugin:67: Loaded plugin:
> > > > > > /usr/lib/vpp_api_test_plugins/flowprobe_test_plugin.so
> > > > > > load_one_plugin:67: Loaded plugin:
> > > > > > /usr/lib/vpp_api_test_plugins/dpdk_test_plugin.so
> > > > > > load_one_plugin:67: Loaded plugin:
> > > > > > /usr/lib/vpp_api_test_plugins/ioam_export_test_plugin.so
> > > > > > load_one_plugin:67: Loaded plugin:
> > > > > > /usr/lib/vpp_api_test_plugins/cdp_test_plugin.so
> > > > > > load_one_plugin:67: Loaded plugin:
> > > > > > /usr/lib/vpp_api_test_plugins/kubeproxy_test_plugin.so
> > > > > > load_one_plugin:67: Loaded plugin:
> > > > > > /usr/lib/vpp_api_test_plugins/ioam_vxlan_gpe_test_plugin.so
> > > > > > load_one_plugin:67: Loaded plugin:
> > > > > > /usr/lib/vpp_api_test_plugins/stn_test_plugin.so
> > > > > > load_one_plugin:67: Loaded plugin:
> > > > > > /usr/lib/vpp_api_test_plugins/acl_test_plugin.so
> > > > > > load_one_plugin:67: Loaded plugin:
> > > > > > /usr/lib/vpp_api_test_plugins/lb_test_plugin.so
> > > > > > load_one_plugin:67: Loaded plugin:
> > > > > > /usr/lib/vpp_api_test_plugins/lacp_test_plugin.so
> > > > > > load_one_plugin:67: Loaded plugin:
> > > > > > /usr/lib/vpp_api_test_plugins/ioam_pot_test_plugin.so
> > > > > > load_one_plugin:67: Loaded plugin:
> > > > > > /usr/lib/vpp_api_test_plugins/pppoe_test_plugin.so
> > > > > > load_one_plugin:67: Loaded plugin:
> > > > > > /usr/lib/vpp_api_test_plugins/udp_ping_test_plugin.so
> > > > > > load_one_plugin:67: Loaded plugin:
> > > > > > /usr/lib/vpp_api_test_plugins/vxlan_gpe_ioam_export_test_plugin.so
> > > > > > load_one_plugin:67: Loaded plugin:
> > > > > > /usr/lib/vpp_api_test_plugins/memif_test_plugin.so
> > > > > > load_one_plugin:67: Loaded plugin:
> > > > > > /usr/lib/vpp_api_test_plugins/gtpu_test_plugin.so
> > > > > > load_one_plugin:67: Loaded plugin:
> > > > > > /usr/lib/vpp_api_test_plugins/ioam_trace_test_plugin.so
> > > > > > load_one_plugin:67: Loaded plugin:
> > > > > > /usr/lib/vpp_api_test_plugins/nat_test_plugin.so
> > > > > > dpdk_config:1271: EAL init args: -c 1 -n 4 --huge-dir
> > > > > > /run/vpp/hugepages --file-prefix vpp -w 0000:00:06.0 -w 0000:00:07.0
> > > > > > --master-lcore 0 --socket-mem 512
> > > > > > EAL: No free hugepages reported in hugepages-1048576kB
> > > > > > EAL: VFIO support initialized
> > > > > > EAL: WARNING: cpu flags constant_tsc=yes nonstop_tsc=no -> using
> > > > > > unreliable clock cycles !
> > > > > > EAL: Invalid NUMA socket, default to 0
> > > > > > EAL: Invalid NUMA socket, default to 0
> > > > > > DPDK physical memory layout:
> > > > > > Segment 0: IOVA:0x33400000, len:25165824, virt:0x7ff739c00000,
> > > > > > socket_id:0, hugepage_sz:2097152, nchannel:0, nrank:0
> > > > > > Segment 1: IOVA:0x34e00000, len:2097152, virt:0x7ff739800000,
> > > > > > socket_id:0, hugepage_sz:2097152, nchannel:0, nrank:0
> > > > > > Segment 2: IOVA:0x87800000, len:509607936, virt:0x7ff707e00000,
> > > > > > socket_id:0, hugepage_sz:2097152, nchannel:0, nrank:0
> > > > > > dpdk_pool_create:550: ioctl(VFIO_IOMMU_MAP_DMA) pool
> > > > > > 'dpdk_mbuf_pool_socket0': Invalid argument (errno 22)
> > > > > > 0: dpdk_ipsec_process:1012: not enough DPDK crypto resources,
> > > > > > default to OpenSSL
> > > > > > 0: dpdk_lib_init:234: DPDK drivers found 2 ports...
> > > > > > _______ _ _ _____ ___
> > > > > > __/ __/ _ \ (_)__ | | / / _ \/ _ \
> > > > > > _/ _// // / / / _ \ | |/ / ___/ ___/
> > > > > > /_/ /____(_)_/\___/ |___/_/ /_/
> > > > > >
> > > > > > DBGvpp#
> > > > > >
> > > > > >
> > > > > >
> > > > > --
> > > > > Marco V
> > > > >
> > > > >
> > > > > SUSE LINUX GmbH | GF: Felix Imendörffer, Jane Smithard, Graham Norton
> > > > > HRB 21284 (AG Nürnberg) Maxfeldstr. 5, D-90409, Nürnberg
> > > >
> > > >
> > > >
> > > --
> > > Marco V
> > >
> > >
> > > SUSE LINUX GmbH | GF: Felix Imendörffer, Jane Smithard, Graham Norton
> > > HRB 21284 (AG Nürnberg) Maxfeldstr. 5, D-90409, Nürnberg
> >
> >
> >
> > --
> > Moon-Sang Lee, SW Engineer
> > Email: sang0...@gmail.com
> > Wisdom begins in wonder. *Socrates*
> >
> >
> >
> >
>
>
>
--
Marco V
SUSE LINUX GmbH | GF: Felix Imendörffer, Jane Smithard, Graham Norton
HRB 21284 (AG Nürnberg) Maxfeldstr. 5, D-90409, Nürnberg
