On Tue, 2018-04-10 at 19:33 +0900, Moon-Sang Lee wrote: > Thanks for your interest, Marco. > I follows the intel guideline, "As an SR-IOV VF network adapter using a KVM > virtual network pool of adapters" > from https://software.intel.com/en-us/articles/configure-sr-iov-network-virtua > l-functions-in-linux-kvm. > > In summary, I modprobe ixgbe on host side and creates one VF per PF. > When I start VM using virsh, libvirt binds the VF to vfio-pci in host side. > After VM finishes booting, I login to VM and bind the VF to igb_uio using > dpdk-dev command. > (i.e. only igb_uio works and other drivers like uio_pci_generic and vfio-pci > fail to bind VF in VM side.) Yes, that's expected. If you want to use vfio-pci in the VM you'll need to enable the "no-iommu":# echo 1 > /sys/module/vfio/parameters/enable_unsafe_noiommu_mode > I don't edit startup.conf in VM, but I just bind the VF to dpdk-compatible > driver, igb_uio, inside VM. > With above configuration, I can bind the VF to guest kernel driver, ixgbevf, > and also to DPDK PMD, igb_uio. > As a result, I can run VPP without DPDK using ixgbevf, and also DPDK > applications using igb_uio. > (i.e. I successfully runs l2fwd/l3fwd sample applications inside VM, so I > guess VF biding has no problem.) > > However, I cannot run VPP with DPDK and I suspect hugepage is related to this > problem as shown in my VPP log. So, what does the command "cat /proc/meminfo |grep HugePages_" shows? > Every packet sent from the VM is traversed via VF to the opposite side, a > pktgen server. > And pktgen replies to those packets, but VM does not receive those replies. > (i.e. I ping to pktgen port from VM, where host server port is directly linked > to pktgen server port.) > > Here is my vpp-config script. > I test ping after running this script. > > #!/bin/sh > > vppctl enable tap-inject > vppctl create loopback interface > vppctl set interface state loop0 up > > vppctl show interface > vppctl show hardware > vppctl set interface state VirtualFunctionEthernet0/6/0 up > vppctl set interface state VirtualFunctionEthernet0/7/0 up > vppctl set interface ip address loop0 2.2.2.2/32 > vppctl set interface ip address VirtualFunctionEthernet0/6/0 192.168.0.1/24 > vppctl set interface ip address VirtualFunctionEthernet0/7/0 192.168.1.1/24 > vppctl show interface address > vppctl show tap-inject > > ip addr add 2.2.2.2/32 dev vpp0 > ip addr add 192.168.0.1/24 dev vpp1 > ip addr add 192.168.1.1/24 dev vpp2 > ip link set dev vpp0 up > ip link set dev vpp1 up > ip link set dev vpp2 up > > > > On Tue, Apr 10, 2018 at 4:51 PM, Marco Varlese <mvarl...@suse.de> wrote: > > On Mon, 2018-04-09 at 22:53 +0900, Moon-Sang Lee wrote: > > > I've configured a VM with KVM, and the VM is intended to run VPP with > > > DPDK. > > > In particular, the VM is connected to one of VFs. (i.e. SR-IOV) > > > I can run DPDK sample applications,including l2fwd and l3fwd, in the VM, > > > therefore I guess VM is successfully connected to the > > > outside-world(pktgen > > > server) via VFs. > > > > > > However, I cannot receive a packet when I run VPP/DPDK. > > > I can see TX packets from the VM on the opposite side, pktgen server, > > > but the VM does not receive any reply from pktgen server which reports > > > RX/TX packet count. > > > (i.e. arping/ping from VM arrives in pktgen, but the reply from pktgen is > > > not received in VM.) > > > I found some strange log messages regarding vpp launching as below. > > > > > > I appreciate for any comment. > > > Thanks in advance... > > > > > > - Host NIC: Intel 82599 10G NIC (i.e. VF binding with vfio-pci) > > > - VM: 1 socket 4 vCPU > > > - VPP: 18.04 > > > - DPDK binding: igb_uio > > It isn't clear to me who manages the PF in the host and how you created the > > VFs (kernel module or via DPDK binding)? > > > > Second, what do you mean by DPDK binding in the last line above? > > Is that what you have configured in startup.conf in the VM for VPP to be > > used? > > > > If so, what is the different between VF binding and DPDK binding in your > > short setup summary above? I'm confused by reading vfio-pci in one place and > > then igb_uio later on. > > > > Can you provide us with the startup.conf you have in the VM? > > > > Finally, if you are interested in using vfio-pci then you'll need to have > > the no-IOMMU enabled otherwise you can't use VFIO in the VM... > > probably, the easiest would be to use igb_uio everywhere... > > > > > root@xenial-vpp-frr:~# vpp -c /etc/vpp/startup.conf > > > vlib_plugin_early_init:359: plugin path /usr/lib/vpp_plugins > > > load_one_plugin:187: Loaded plugin: acl_plugin.so (Access Control Lists) > > > load_one_plugin:187: Loaded plugin: avf_plugin.so (Intel Adaptive Virtual > > > Function (AVF) Device Plugin) > > > load_one_plugin:189: Loaded plugin: cdp_plugin.so > > > load_one_plugin:187: Loaded plugin: dpdk_plugin.so (Data Plane Development > > > Kit (DPDK)) > > > load_one_plugin:187: Loaded plugin: flowprobe_plugin.so (Flow per Packet) > > > load_one_plugin:187: Loaded plugin: gbp_plugin.so (Group Based Policy) > > > load_one_plugin:187: Loaded plugin: gtpu_plugin.so (GTPv1-U) > > > load_one_plugin:187: Loaded plugin: igmp_plugin.so (IGMP messaging) > > > load_one_plugin:187: Loaded plugin: ila_plugin.so (Identifier-locator > > > addressing for IPv6) > > > load_one_plugin:187: Loaded plugin: ioam_plugin.so (Inbound OAM) > > > load_one_plugin:117: Plugin disabled (default): ixge_plugin.so > > > load_one_plugin:187: Loaded plugin: kubeproxy_plugin.so (kube-proxy data > > > plane) > > > load_one_plugin:187: Loaded plugin: l2e_plugin.so (L2 Emulation) > > > load_one_plugin:187: Loaded plugin: lacp_plugin.so (Link Aggregation > > > Control Protocol) > > > load_one_plugin:187: Loaded plugin: lb_plugin.so (Load Balancer) > > > load_one_plugin:187: Loaded plugin: memif_plugin.so (Packet Memory > > > Interface (experimetal)) > > > load_one_plugin:187: Loaded plugin: nat_plugin.so (Network Address > > > Translation) > > > load_one_plugin:187: Loaded plugin: pppoe_plugin.so (PPPoE) > > > load_one_plugin:187: Loaded plugin: router.so (router) > > > load_one_plugin:187: Loaded plugin: srv6ad_plugin.so (Dynamic SRv6 proxy) > > > load_one_plugin:187: Loaded plugin: srv6am_plugin.so (Masquerading SRv6 > > > proxy) > > > load_one_plugin:187: Loaded plugin: srv6as_plugin.so (Static SRv6 proxy) > > > load_one_plugin:187: Loaded plugin: stn_plugin.so (VPP Steals the NIC for > > > Container integration) > > > load_one_plugin:187: Loaded plugin: tlsmbedtls_plugin.so (mbedtls based > > > TLS Engine) > > > load_one_plugin:187: Loaded plugin: tlsopenssl_plugin.so (openssl based > > > TLS Engine) > > > load_one_plugin:67: Loaded plugin: > > > /usr/lib/vpp_api_test_plugins/flowprobe_test_plugin.so > > > load_one_plugin:67: Loaded plugin: > > > /usr/lib/vpp_api_test_plugins/dpdk_test_plugin.so > > > load_one_plugin:67: Loaded plugin: > > > /usr/lib/vpp_api_test_plugins/ioam_export_test_plugin.so > > > load_one_plugin:67: Loaded plugin: > > > /usr/lib/vpp_api_test_plugins/cdp_test_plugin.so > > > load_one_plugin:67: Loaded plugin: > > > /usr/lib/vpp_api_test_plugins/kubeproxy_test_plugin.so > > > load_one_plugin:67: Loaded plugin: > > > /usr/lib/vpp_api_test_plugins/ioam_vxlan_gpe_test_plugin.so > > > load_one_plugin:67: Loaded plugin: > > > /usr/lib/vpp_api_test_plugins/stn_test_plugin.so > > > load_one_plugin:67: Loaded plugin: > > > /usr/lib/vpp_api_test_plugins/acl_test_plugin.so > > > load_one_plugin:67: Loaded plugin: > > > /usr/lib/vpp_api_test_plugins/lb_test_plugin.so > > > load_one_plugin:67: Loaded plugin: > > > /usr/lib/vpp_api_test_plugins/lacp_test_plugin.so > > > load_one_plugin:67: Loaded plugin: > > > /usr/lib/vpp_api_test_plugins/ioam_pot_test_plugin.so > > > load_one_plugin:67: Loaded plugin: > > > /usr/lib/vpp_api_test_plugins/pppoe_test_plugin.so > > > load_one_plugin:67: Loaded plugin: > > > /usr/lib/vpp_api_test_plugins/udp_ping_test_plugin.so > > > load_one_plugin:67: Loaded plugin: > > > /usr/lib/vpp_api_test_plugins/vxlan_gpe_ioam_export_test_plugin.so > > > load_one_plugin:67: Loaded plugin: > > > /usr/lib/vpp_api_test_plugins/memif_test_plugin.so > > > load_one_plugin:67: Loaded plugin: > > > /usr/lib/vpp_api_test_plugins/gtpu_test_plugin.so > > > load_one_plugin:67: Loaded plugin: > > > /usr/lib/vpp_api_test_plugins/ioam_trace_test_plugin.so > > > load_one_plugin:67: Loaded plugin: > > > /usr/lib/vpp_api_test_plugins/nat_test_plugin.so > > > dpdk_config:1271: EAL init args: -c 1 -n 4 --huge-dir /run/vpp/hugepages > > > --file-prefix vpp -w 0000:00:06.0 -w 0000:00:07.0 --master-lcore 0 -- > > > socket-mem 512 > > > EAL: No free hugepages reported in hugepages-1048576kB > > > EAL: VFIO support initialized > > > EAL: WARNING: cpu flags constant_tsc=yes nonstop_tsc=no -> using > > > unreliable clock cycles ! > > > EAL: Invalid NUMA socket, default to 0 > > > EAL: Invalid NUMA socket, default to 0 > > > DPDK physical memory layout: > > > Segment 0: IOVA:0x33400000, len:25165824, virt:0x7ff739c00000, > > > socket_id:0, hugepage_sz:2097152, nchannel:0, nrank:0 > > > Segment 1: IOVA:0x34e00000, len:2097152, virt:0x7ff739800000, socket_id:0, > > > hugepage_sz:2097152, nchannel:0, nrank:0 > > > Segment 2: IOVA:0x87800000, len:509607936, virt:0x7ff707e00000, > > > socket_id:0, hugepage_sz:2097152, nchannel:0, nrank:0 > > > dpdk_pool_create:550: ioctl(VFIO_IOMMU_MAP_DMA) pool > > > 'dpdk_mbuf_pool_socket0': Invalid argument (errno 22) > > > 0: dpdk_ipsec_process:1012: not enough DPDK crypto resources, default to > > > OpenSSL > > > 0: dpdk_lib_init:234: DPDK drivers found 2 ports... > > > _______ _ _ _____ ___ > > > __/ __/ _ \ (_)__ | | / / _ \/ _ \ > > > _/ _// // / / / _ \ | |/ / ___/ ___/ > > > /_/ /____(_)_/\___/ |___/_/ /_/ > > > > > > DBGvpp# > > > > > > > > > > > -- > > Marco V > > > > > > SUSE LINUX GmbH | GF: Felix Imendörffer, Jane Smithard, Graham Norton > > HRB 21284 (AG Nürnberg) Maxfeldstr. 5, D-90409, Nürnberg > > > -- Marco V
SUSE LINUX GmbH | GF: Felix Imendörffer, Jane Smithard, Graham Norton HRB 21284 (AG Nürnberg) Maxfeldstr. 5, D-90409, Nürnberg
