On Tue, Apr 10, 2018 at 8:24 PM, Marco Varlese <mvarl...@suse.de> wrote:
> On Tue, 2018-04-10 at 19:33 +0900, Moon-Sang Lee wrote: > > > Thanks for your interest, Marco. > > I follows the intel guideline, "As an SR-IOV VF network adapter using a > KVM virtual network pool of adapters" > from https://software.intel.com/en-us/articles/configure- > sr-iov-network-virtual-functions-in-linux-kvm. > > In summary, I modprobe ixgbe on host side and creates one VF per PF. > When I start VM using virsh, libvirt binds the VF to vfio-pci in host side. > After VM finishes booting, I login to VM and bind the VF to igb_uio using > dpdk-dev command. > (i.e. only igb_uio works and other drivers like uio_pci_generic and > vfio-pci fail to bind VF in VM side.) > > Yes, that's expected. > If you want to use vfio-pci in the VM you'll need to enable the "no-iommu": > # echo 1 > /sys/module/vfio/parameters/enable_unsafe_noiommu_mode > well, I cannot find such file in host nor VM. > > > I don't edit startup.conf in VM, but I just bind the VF to dpdk-compatible > driver, igb_uio, inside VM. > With above configuration, I can bind the VF to guest kernel driver, > ixgbevf, and also to DPDK PMD, igb_uio. > As a result, I can run VPP without DPDK using ixgbevf, and also DPDK > applications using igb_uio. > (i.e. I successfully runs l2fwd/l3fwd sample applications inside VM, so I > guess VF biding has no problem.) > > However, I cannot run VPP with DPDK and I suspect hugepage is related to > this problem as shown in my VPP log. > > So, what does the command "cat /proc/meminfo |grep HugePages_" shows? > well, 'cat /proc/meminfo |grep HugePages' seems ok even though I cannot find any rtemap files neither in /dev/hugepages nor /run/vpp/hugepages. When I run dpdk application, I can see rtemap files in /dev/hugepages. HugePages_Total: 1024 HugePages_Free: 972 HugePages_Rsvd: 0 HugePages_Surp: 0 Hugepagesize: 2048 kB FYI. Apr 10 05:12:35 ubuntu1604 /usr/bin/vpp[1720]: dpdk_config:1271: EAL init args: -c 1 -n 4 --huge-dir /run/vpp/hugepages --file-prefix vpp -b 0000:00:05.0 -b 0000:00:06.0 -b 0000:00:07.0 --master-lcore 0 --socket-mem 64 > > Every packet sent from the VM is traversed via VF to the opposite side, a > pktgen server. > And pktgen replies to those packets, but VM does not receive those replies. > (i.e. I ping to pktgen port from VM, where host server port is directly > linked to pktgen server port.) > > Here is my vpp-config script. > I test ping after running this script. > > #!/bin/sh > > vppctl enable tap-inject > vppctl create loopback interface > vppctl set interface state loop0 up > > vppctl show interface > vppctl show hardware > vppctl set interface state VirtualFunctionEthernet0/6/0 up > vppctl set interface state VirtualFunctionEthernet0/7/0 up > vppctl set interface ip address loop0 2.2.2.2/32 > vppctl set interface ip address VirtualFunctionEthernet0/6/0 > 192.168.0.1/24 > vppctl set interface ip address VirtualFunctionEthernet0/7/0 > 192.168.1.1/24 > vppctl show interface address > vppctl show tap-inject > > ip addr add 2.2.2.2/32 dev vpp0 > ip addr add 192.168.0.1/24 dev vpp1 > ip addr add 192.168.1.1/24 dev vpp2 > ip link set dev vpp0 up > ip link set dev vpp1 up > ip link set dev vpp2 up > > > On Tue, Apr 10, 2018 at 4:51 PM, Marco Varlese <mvarl...@suse.de> wrote: > > On Mon, 2018-04-09 at 22:53 +0900, Moon-Sang Lee wrote: > > > I've configured a VM with KVM, and the VM is intended to run VPP with DPDK. > In particular, the VM is connected to one of VFs. (i.e. SR-IOV) > I can run DPDK sample applications,including l2fwd and l3fwd, in the VM, > therefore I guess VM is successfully connected to the outside-world(pktgen > server) via VFs. > > However, I cannot receive a packet when I run VPP/DPDK. > I can see TX packets from the VM on the opposite side, pktgen server, > but the VM does not receive any reply from pktgen server which reports > RX/TX packet count. > (i.e. arping/ping from VM arrives in pktgen, but the reply from pktgen is > not received in VM.) > I found some strange log messages regarding vpp launching as below. > > I appreciate for any comment. > Thanks in advance... > > - Host NIC: Intel 82599 10G NIC (i.e. VF binding with vfio-pci) > - VM: 1 socket 4 vCPU > - VPP: 18.04 > - DPDK binding: igb_uio > > It isn't clear to me who manages the PF in the host and how you created > the VFs (kernel module or via DPDK binding)? > > Second, what do you mean by DPDK binding in the last line above? > Is that what you have configured in startup.conf in the VM for VPP to be > used? > > If so, what is the different between VF binding and DPDK binding in your > short setup summary above? I'm confused by reading vfio-pci in one place > and then igb_uio later on. > > Can you provide us with the startup.conf you have in the VM? > > Finally, if you are interested in using vfio-pci then you'll need to have > the no-IOMMU enabled otherwise you can't use VFIO in the VM... > probably, the easiest would be to use igb_uio everywhere... > > > root@xenial-vpp-frr:~# vpp -c /etc/vpp/startup.conf > vlib_plugin_early_init:359: plugin path /usr/lib/vpp_plugins > load_one_plugin:187: Loaded plugin: acl_plugin.so (Access Control Lists) > load_one_plugin:187: Loaded plugin: avf_plugin.so (Intel Adaptive Virtual > Function (AVF) Device Plugin) > load_one_plugin:189: Loaded plugin: cdp_plugin.so > load_one_plugin:187: Loaded plugin: dpdk_plugin.so (Data Plane Development > Kit (DPDK)) > load_one_plugin:187: Loaded plugin: flowprobe_plugin.so (Flow per Packet) > load_one_plugin:187: Loaded plugin: gbp_plugin.so (Group Based Policy) > load_one_plugin:187: Loaded plugin: gtpu_plugin.so (GTPv1-U) > load_one_plugin:187: Loaded plugin: igmp_plugin.so (IGMP messaging) > load_one_plugin:187: Loaded plugin: ila_plugin.so (Identifier-locator > addressing for IPv6) > load_one_plugin:187: Loaded plugin: ioam_plugin.so (Inbound OAM) > load_one_plugin:117: Plugin disabled (default): ixge_plugin.so > load_one_plugin:187: Loaded plugin: kubeproxy_plugin.so (kube-proxy data > plane) > load_one_plugin:187: Loaded plugin: l2e_plugin.so (L2 Emulation) > load_one_plugin:187: Loaded plugin: lacp_plugin.so (Link Aggregation > Control Protocol) > load_one_plugin:187: Loaded plugin: lb_plugin.so (Load Balancer) > load_one_plugin:187: Loaded plugin: memif_plugin.so (Packet Memory > Interface (experimetal)) > load_one_plugin:187: Loaded plugin: nat_plugin.so (Network Address > Translation) > load_one_plugin:187: Loaded plugin: pppoe_plugin.so (PPPoE) > load_one_plugin:187: Loaded plugin: router.so (router) > load_one_plugin:187: Loaded plugin: srv6ad_plugin.so (Dynamic SRv6 proxy) > load_one_plugin:187: Loaded plugin: srv6am_plugin.so (Masquerading SRv6 > proxy) > load_one_plugin:187: Loaded plugin: srv6as_plugin.so (Static SRv6 proxy) > load_one_plugin:187: Loaded plugin: stn_plugin.so (VPP Steals the NIC for > Container integration) > load_one_plugin:187: Loaded plugin: tlsmbedtls_plugin.so (mbedtls based > TLS Engine) > load_one_plugin:187: Loaded plugin: tlsopenssl_plugin.so (openssl based > TLS Engine) > load_one_plugin:67: Loaded plugin: /usr/lib/vpp_api_test_plugins/ > flowprobe_test_plugin.so > load_one_plugin:67: Loaded plugin: /usr/lib/vpp_api_test_plugins/ > dpdk_test_plugin.so > load_one_plugin:67: Loaded plugin: /usr/lib/vpp_api_test_plugins/ > ioam_export_test_plugin.so > load_one_plugin:67: Loaded plugin: /usr/lib/vpp_api_test_plugins/ > cdp_test_plugin.so > load_one_plugin:67: Loaded plugin: /usr/lib/vpp_api_test_plugins/ > kubeproxy_test_plugin.so > load_one_plugin:67: Loaded plugin: /usr/lib/vpp_api_test_plugins/ > ioam_vxlan_gpe_test_plugin.so > load_one_plugin:67: Loaded plugin: /usr/lib/vpp_api_test_plugins/ > stn_test_plugin.so > load_one_plugin:67: Loaded plugin: /usr/lib/vpp_api_test_plugins/ > acl_test_plugin.so > load_one_plugin:67: Loaded plugin: /usr/lib/vpp_api_test_plugins/ > lb_test_plugin.so > load_one_plugin:67: Loaded plugin: /usr/lib/vpp_api_test_plugins/ > lacp_test_plugin.so > load_one_plugin:67: Loaded plugin: /usr/lib/vpp_api_test_plugins/ > ioam_pot_test_plugin.so > load_one_plugin:67: Loaded plugin: /usr/lib/vpp_api_test_plugins/ > pppoe_test_plugin.so > load_one_plugin:67: Loaded plugin: /usr/lib/vpp_api_test_plugins/ > udp_ping_test_plugin.so > load_one_plugin:67: Loaded plugin: /usr/lib/vpp_api_test_plugins/ > vxlan_gpe_ioam_export_test_plugin.so > load_one_plugin:67: Loaded plugin: /usr/lib/vpp_api_test_plugins/ > memif_test_plugin.so > load_one_plugin:67: Loaded plugin: /usr/lib/vpp_api_test_plugins/ > gtpu_test_plugin.so > load_one_plugin:67: Loaded plugin: /usr/lib/vpp_api_test_plugins/ > ioam_trace_test_plugin.so > load_one_plugin:67: Loaded plugin: /usr/lib/vpp_api_test_plugins/ > nat_test_plugin.so > dpdk_config:1271: EAL init args: -c 1 -n 4 --huge-dir /run/vpp/hugepages > --file-prefix vpp -w 0000:00:06.0 -w 0000:00:07.0 --master-lcore 0 > --socket-mem 512 > EAL: No free hugepages reported in hugepages-1048576kB > EAL: VFIO support initialized > EAL: WARNING: cpu flags constant_tsc=yes nonstop_tsc=no -> using > unreliable clock cycles ! > EAL: Invalid NUMA socket, default to 0 > EAL: Invalid NUMA socket, default to 0 > DPDK physical memory layout: > Segment 0: IOVA:0x33400000, len:25165824, virt:0x7ff739c00000, > socket_id:0, hugepage_sz:2097152, nchannel:0, nrank:0 > Segment 1: IOVA:0x34e00000, len:2097152, virt:0x7ff739800000, socket_id:0, > hugepage_sz:2097152, nchannel:0, nrank:0 > Segment 2: IOVA:0x87800000, len:509607936, virt:0x7ff707e00000, > socket_id:0, hugepage_sz:2097152, nchannel:0, nrank:0 > dpdk_pool_create:550: ioctl(VFIO_IOMMU_MAP_DMA) pool > 'dpdk_mbuf_pool_socket0': Invalid argument (errno 22) > 0: dpdk_ipsec_process:1012: not enough DPDK crypto resources, default to > OpenSSL > 0: dpdk_lib_init:234: DPDK drivers found 2 ports... > _______ _ _ _____ ___ > __/ __/ _ \ (_)__ | | / / _ \/ _ \ > _/ _// // / / / _ \ | |/ / ___/ ___/ > /_/ /____(_)_/\___/ |___/_/ /_/ > > DBGvpp# > > > > -- > > Marco V > > SUSE LINUX GmbH | GF: Felix Imendörffer, Jane Smithard, Graham Norton > HRB 21284 (AG Nürnberg) Maxfeldstr. 5, D-90409, Nürnberg > <https://maps.google.com/?q=Maxfeldstr.+5,+D-90409,+N%C3%BCrnberg&entry=gmail&source=g> > > > > > -- > > Marco V > > SUSE LINUX GmbH | GF: Felix Imendörffer, Jane Smithard, Graham Norton > HRB 21284 (AG Nürnberg) Maxfeldstr. 5, D-90409, Nürnberg > <https://maps.google.com/?q=Maxfeldstr.+5,+D-90409,+N%C3%BCrnberg&entry=gmail&source=g> > -- Moon-Sang Lee, SW Engineer Email: sang0...@gmail.com Wisdom begins in wonder. *Socrates*
