On Wed, 2018-04-11 at 17:10 +0900, Moon-Sang Lee wrote:
> I first bind VF to igb_uio in /etc/rc.local of VM, and I guess VPP take the
> config.
> I did not edited startup.conf yesterday, and I edit it today as below.
>
> cpu {
> workers 2
> }
>
> dpdk {
> dev 0000:00:08.0
> dev 0000:00:09.0
> uio-driver igb_uio
> }
>
> Anyway, I got to succeed to measure VPP/DPDK performance with pktgen.
Glad, you managed to.
> Here is my guess about running VPP/DPDK with pktgen.
> - I think VF's MTU should match to that of PF. (i.e. PF 1500B vs VF 9220B
> was my issue, I guess.)
Ok, but yesterday VPP wasn't getting initialized at all... so I believe you
managed to get your port initialized by correctly setting the mapping to igb_uio
in startup.conf (which I understand you did today...)
> - On host side, I have ping using PF to pktgen port and this enables VF on
> VM side communicate with pktgen.
>
> Thank you for your kind discussion, Marco.
You quite welcome. Glad you got this going...
>
>
> On Wed, Apr 11, 2018 at 4:49 PM, Marco Varlese <mvarl...@suse.de> wrote:
> > If I remember correctly, you mentioned in early email that you do _NOT_
> > modify the startup.conf file for VPP. Can you confirm that?
> >
> > By looking at the startup.conf file coming with VPP v18.04 the default
> > mechanism driver used is vfio-pci. That as mentioned before (and found out
> > by you too) won't work.
> > Also, by default the DPDK section is completely commented out so all the
> > defaults are being used...
> >
> > Can you please take a look at your startup.conf file and make sure to use
> > igb_uio by specifying "uio-driver igb_uio"
> >
> > And, first of all, make sure the dpdk section is not commented out?
> >
> > I think something like the below should help you to get going...
> >
> > dpdk {
> > uio-driver igb_uio
> > }
> >
> >
> > - Marco
> > On Wed, 2018-04-11 at 16:12 +0900, Moon-Sang Lee wrote:
> > > I'm back with a little progress.
> > > Anyway, I found that vpp interface has MTU size as 9220B.
> > > And I just tried to change it to 1500B as below.
> > > # vppctl set interface mtu 1500 VirtualFunctionEthernet0/8/0
> > >
> > > It works!
> > > My VM with VPP/DPDK ping/pong to outside server via VF of Intel 82599 10G
> > > NIC.
> > > But, I could not understand why MTU configuration affect to simple ping
> > > test.
> > >
> > > In addition, I need to measure this VM's performance with pktgen-dpdk
> > > which has no MTU option as far as I know.
> > > When I shoot packets from pktgen, VM does not receive packets nor ping
> > > does not work again.
> > > (i.e. pktgen and VM are directly connected with 10G link.)
> > >
> > > I appreciate any comment.
> > >
> > >
> > >
> > > On Tue, Apr 10, 2018 at 9:16 PM, Moon-Sang Lee <sang0...@gmail.com> wrote:
> > > > On Tue, Apr 10, 2018 at 8:24 PM, Marco Varlese <mvarl...@suse.de> wrote:
> > > > > On Tue, 2018-04-10 at 19:33 +0900, Moon-Sang Lee wrote:
> > > > > > Thanks for your interest, Marco.
> > > > > > I follows the intel guideline, "As an SR-IOV VF network adapter
> > > > > > using a KVM virtual network pool of adapters"
> > > > > > from https://software.intel.com/en-us/articles/configure-sr-iov-netw
> > > > > > ork-virtual-functions-in-linux-kvm.
> > > > > >
> > > > > > In summary, I modprobe ixgbe on host side and creates one VF per
> > > > > > PF.
> > > > > > When I start VM using virsh, libvirt binds the VF to vfio-pci in
> > > > > > host side.
> > > > > > After VM finishes booting, I login to VM and bind the VF to igb_uio
> > > > > > using dpdk-dev command.
> > > > > > (i.e. only igb_uio works and other drivers like uio_pci_generic and
> > > > > > vfio-pci fail to bind VF in VM side.)
> > > > > Yes, that's expected.
> > > > > If you want to use vfio-pci in the VM you'll need to enable the "no-
> > > > > iommu":
> > > > > # echo 1 > /sys/module/vfio/parameters/enable_unsafe_noiommu_mode
> > > >
> > > > well, I cannot find such file in host nor VM.
> > > >
> > > > > > I don't edit startup.conf in VM, but I just bind the VF to dpdk-
> > > > > > compatible driver, igb_uio, inside VM.
> > > > > > With above configuration, I can bind the VF to guest kernel driver,
> > > > > > ixgbevf, and also to DPDK PMD, igb_uio.
> > > > > > As a result, I can run VPP without DPDK using ixgbevf, and also
> > > > > > DPDK applications using igb_uio.
> > > > > > (i.e. I successfully runs l2fwd/l3fwd sample applications inside VM,
> > > > > > so I guess VF biding has no problem.)
> > > > > >
> > > > > > However, I cannot run VPP with DPDK and I suspect hugepage is
> > > > > > related to this problem as shown in my VPP log.
> > > > > So, what does the command "cat /proc/meminfo |grep HugePages_" shows?
> > > >
> > > > well, 'cat /proc/meminfo |grep HugePages' seems ok even though I cannot
> > > > find any rtemap files neither in /dev/hugepages nor /run/vpp/hugepages.
> > > > When I run dpdk application, I can see rtemap files in /dev/hugepages.
> > > >
> > > > HugePages_Total: 1024
> > > > HugePages_Free: 972
> > > > HugePages_Rsvd: 0
> > > > HugePages_Surp: 0
> > > > Hugepagesize: 2048 kB
> > > >
> > > > FYI.Apr 10 05:12:35 ubuntu1604 /usr/bin/vpp[1720]: dpdk_config:1271: EAL
> > > > init args: -c 1 -n 4 --huge-dir /run/vpp/hugepages --file-prefix vpp -b
> > > > 0000:00:05.0 -b 0000:00:06.0 -b 0000:00:07.0 --master-lcore 0 --socket-
> > > > mem 64
> > > >
> > > >
> > > > > > Every packet sent from the VM is traversed via VF to the opposite
> > > > > > side, a pktgen server.
> > > > > > And pktgen replies to those packets, but VM does not receive those
> > > > > > replies.
> > > > > > (i.e. I ping to pktgen port from VM, where host server port is
> > > > > > directly linked to pktgen server port.)
> > > > > >
> > > > > > Here is my vpp-config script.
> > > > > > I test ping after running this script.
> > > > > >
> > > > > > #!/bin/sh
> > > > > >
> > > > > > vppctl enable tap-inject
> > > > > > vppctl create loopback interface
> > > > > > vppctl set interface state loop0 up
> > > > > >
> > > > > > vppctl show interface
> > > > > > vppctl show hardware
> > > > > > vppctl set interface state VirtualFunctionEthernet0/6/0 up
> > > > > > vppctl set interface state VirtualFunctionEthernet0/7/0 up
> > > > > > vppctl set interface ip address loop0 2.2.2.2/32
> > > > > > vppctl set interface ip address VirtualFunctionEthernet0/6/0
> > > > > > 192.168.0.1/24
> > > > > > vppctl set interface ip address VirtualFunctionEthernet0/7/0
> > > > > > 192.168.1.1/24
> > > > > > vppctl show interface address
> > > > > > vppctl show tap-inject
> > > > > >
> > > > > > ip addr add 2.2.2.2/32 dev vpp0
> > > > > > ip addr add 192.168.0.1/24 dev vpp1
> > > > > > ip addr add 192.168.1.1/24 dev vpp2
> > > > > > ip link set dev vpp0 up
> > > > > > ip link set dev vpp1 up
> > > > > > ip link set dev vpp2 up
> > > > > >
> > > > > >
> > > > > >
> > > > > > On Tue, Apr 10, 2018 at 4:51 PM, Marco Varlese <mvarl...@suse.de>
> > > > > > wrote:
> > > > > > > On Mon, 2018-04-09 at 22:53 +0900, Moon-Sang Lee wrote:
> > > > > > > > I've configured a VM with KVM, and the VM is intended to run VPP
> > > > > > > > with DPDK.
> > > > > > > > In particular, the VM is connected to one of VFs. (i.e. SR-IOV)
> > > > > > > > I can run DPDK sample applications,including l2fwd and l3fwd, in
> > > > > > > > the VM,
> > > > > > > > therefore I guess VM is successfully connected to the outside-
> > > > > > > > world(pktgen server) via VFs.
> > > > > > > >
> > > > > > > > However, I cannot receive a packet when I run VPP/DPDK.
> > > > > > > > I can see TX packets from the VM on the opposite side, pktgen
> > > > > > > > server,
> > > > > > > > but the VM does not receive any reply from pktgen server which
> > > > > > > > reports RX/TX packet count.
> > > > > > > > (i.e. arping/ping from VM arrives in pktgen, but the reply from
> > > > > > > > pktgen is not received in VM.)
> > > > > > > > I found some strange log messages regarding vpp launching as
> > > > > > > > below.
> > > > > > > >
> > > > > > > > I appreciate for any comment.
> > > > > > > > Thanks in advance...
> > > > > > > >
> > > > > > > > - Host NIC: Intel 82599 10G NIC (i.e. VF binding with vfio-pci)
> > > > > > > > - VM: 1 socket 4 vCPU
> > > > > > > > - VPP: 18.04
> > > > > > > > - DPDK binding: igb_uio
> > > > > > > It isn't clear to me who manages the PF in the host and how you
> > > > > > > created the VFs (kernel module or via DPDK binding)?
> > > > > > >
> > > > > > > Second, what do you mean by DPDK binding in the last line above?
> > > > > > > Is that what you have configured in startup.conf in the VM for VPP
> > > > > > > to be used?
> > > > > > >
> > > > > > > If so, what is the different between VF binding and DPDK binding
> > > > > > > in your short setup summary above? I'm confused by reading vfio-
> > > > > > > pci in one place and then igb_uio later on.
> > > > > > >
> > > > > > > Can you provide us with the startup.conf you have in the VM?
> > > > > > >
> > > > > > > Finally, if you are interested in using vfio-pci then you'll need
> > > > > > > to have the no-IOMMU enabled otherwise you can't use VFIO in the
> > > > > > > VM...
> > > > > > > probably, the easiest would be to use igb_uio everywhere...
> > > > > > >
> > > > > > > > root@xenial-vpp-frr:~# vpp -c /etc/vpp/startup.conf
> > > > > > > > vlib_plugin_early_init:359: plugin path /usr/lib/vpp_plugins
> > > > > > > > load_one_plugin:187: Loaded plugin: acl_plugin.so (Access
> > > > > > > > Control Lists)
> > > > > > > > load_one_plugin:187: Loaded plugin: avf_plugin.so (Intel
> > > > > > > > Adaptive Virtual Function (AVF) Device Plugin)
> > > > > > > > load_one_plugin:189: Loaded plugin: cdp_plugin.so
> > > > > > > > load_one_plugin:187: Loaded plugin: dpdk_plugin.so (Data Plane
> > > > > > > > Development Kit (DPDK))
> > > > > > > > load_one_plugin:187: Loaded plugin: flowprobe_plugin.so (Flow
> > > > > > > > per Packet)
> > > > > > > > load_one_plugin:187: Loaded plugin: gbp_plugin.so (Group Based
> > > > > > > > Policy)
> > > > > > > > load_one_plugin:187: Loaded plugin: gtpu_plugin.so (GTPv1-U)
> > > > > > > > load_one_plugin:187: Loaded plugin: igmp_plugin.so (IGMP
> > > > > > > > messaging)
> > > > > > > > load_one_plugin:187: Loaded plugin: ila_plugin.so (Identifier-
> > > > > > > > locator addressing for IPv6)
> > > > > > > > load_one_plugin:187: Loaded plugin: ioam_plugin.so (Inbound OAM)
> > > > > > > > load_one_plugin:117: Plugin disabled (default): ixge_plugin.so
> > > > > > > > load_one_plugin:187: Loaded plugin: kubeproxy_plugin.so (kube-
> > > > > > > > proxy data plane)
> > > > > > > > load_one_plugin:187: Loaded plugin: l2e_plugin.so (L2 Emulation)
> > > > > > > > load_one_plugin:187: Loaded plugin: lacp_plugin.so (Link
> > > > > > > > Aggregation Control Protocol)
> > > > > > > > load_one_plugin:187: Loaded plugin: lb_plugin.so (Load Balancer)
> > > > > > > > load_one_plugin:187: Loaded plugin: memif_plugin.so (Packet
> > > > > > > > Memory Interface (experimetal))
> > > > > > > > load_one_plugin:187: Loaded plugin: nat_plugin.so (Network
> > > > > > > > Address Translation)
> > > > > > > > load_one_plugin:187: Loaded plugin: pppoe_plugin.so (PPPoE)
> > > > > > > > load_one_plugin:187: Loaded plugin: router.so (router)
> > > > > > > > load_one_plugin:187: Loaded plugin: srv6ad_plugin.so (Dynamic
> > > > > > > > SRv6 proxy)
> > > > > > > > load_one_plugin:187: Loaded plugin: srv6am_plugin.so
> > > > > > > > (Masquerading SRv6 proxy)
> > > > > > > > load_one_plugin:187: Loaded plugin: srv6as_plugin.so (Static
> > > > > > > > SRv6 proxy)
> > > > > > > > load_one_plugin:187: Loaded plugin: stn_plugin.so (VPP Steals
> > > > > > > > the NIC for Container integration)
> > > > > > > > load_one_plugin:187: Loaded plugin: tlsmbedtls_plugin.so
> > > > > > > > (mbedtls based TLS Engine)
> > > > > > > > load_one_plugin:187: Loaded plugin: tlsopenssl_plugin.so
> > > > > > > > (openssl based TLS Engine)
> > > > > > > > load_one_plugin:67: Loaded plugin:
> > > > > > > > /usr/lib/vpp_api_test_plugins/flowprobe_test_plugin.so
> > > > > > > > load_one_plugin:67: Loaded plugin:
> > > > > > > > /usr/lib/vpp_api_test_plugins/dpdk_test_plugin.so
> > > > > > > > load_one_plugin:67: Loaded plugin:
> > > > > > > > /usr/lib/vpp_api_test_plugins/ioam_export_test_plugin.so
> > > > > > > > load_one_plugin:67: Loaded plugin:
> > > > > > > > /usr/lib/vpp_api_test_plugins/cdp_test_plugin.so
> > > > > > > > load_one_plugin:67: Loaded plugin:
> > > > > > > > /usr/lib/vpp_api_test_plugins/kubeproxy_test_plugin.so
> > > > > > > > load_one_plugin:67: Loaded plugin:
> > > > > > > > /usr/lib/vpp_api_test_plugins/ioam_vxlan_gpe_test_plugin.so
> > > > > > > > load_one_plugin:67: Loaded plugin:
> > > > > > > > /usr/lib/vpp_api_test_plugins/stn_test_plugin.so
> > > > > > > > load_one_plugin:67: Loaded plugin:
> > > > > > > > /usr/lib/vpp_api_test_plugins/acl_test_plugin.so
> > > > > > > > load_one_plugin:67: Loaded plugin:
> > > > > > > > /usr/lib/vpp_api_test_plugins/lb_test_plugin.so
> > > > > > > > load_one_plugin:67: Loaded plugin:
> > > > > > > > /usr/lib/vpp_api_test_plugins/lacp_test_plugin.so
> > > > > > > > load_one_plugin:67: Loaded plugin:
> > > > > > > > /usr/lib/vpp_api_test_plugins/ioam_pot_test_plugin.so
> > > > > > > > load_one_plugin:67: Loaded plugin:
> > > > > > > > /usr/lib/vpp_api_test_plugins/pppoe_test_plugin.so
> > > > > > > > load_one_plugin:67: Loaded plugin:
> > > > > > > > /usr/lib/vpp_api_test_plugins/udp_ping_test_plugin.so
> > > > > > > > load_one_plugin:67: Loaded plugin:
> > > > > > > > /usr/lib/vpp_api_test_plugins/vxlan_gpe_ioam_export_test_plugin.
> > > > > > > > so
> > > > > > > > load_one_plugin:67: Loaded plugin:
> > > > > > > > /usr/lib/vpp_api_test_plugins/memif_test_plugin.so
> > > > > > > > load_one_plugin:67: Loaded plugin:
> > > > > > > > /usr/lib/vpp_api_test_plugins/gtpu_test_plugin.so
> > > > > > > > load_one_plugin:67: Loaded plugin:
> > > > > > > > /usr/lib/vpp_api_test_plugins/ioam_trace_test_plugin.so
> > > > > > > > load_one_plugin:67: Loaded plugin:
> > > > > > > > /usr/lib/vpp_api_test_plugins/nat_test_plugin.so
> > > > > > > > dpdk_config:1271: EAL init args: -c 1 -n 4 --huge-dir
> > > > > > > > /run/vpp/hugepages --file-prefix vpp -w 0000:00:06.0 -w
> > > > > > > > 0000:00:07.0 --master-lcore 0 --socket-mem 512
> > > > > > > > EAL: No free hugepages reported in hugepages-1048576kB
> > > > > > > > EAL: VFIO support initialized
> > > > > > > > EAL: WARNING: cpu flags constant_tsc=yes nonstop_tsc=no -> using
> > > > > > > > unreliable clock cycles !
> > > > > > > > EAL: Invalid NUMA socket, default to 0
> > > > > > > > EAL: Invalid NUMA socket, default to 0
> > > > > > > > DPDK physical memory layout:
> > > > > > > > Segment 0: IOVA:0x33400000, len:25165824, virt:0x7ff739c00000,
> > > > > > > > socket_id:0, hugepage_sz:2097152, nchannel:0, nrank:0
> > > > > > > > Segment 1: IOVA:0x34e00000, len:2097152, virt:0x7ff739800000,
> > > > > > > > socket_id:0, hugepage_sz:2097152, nchannel:0, nrank:0
> > > > > > > > Segment 2: IOVA:0x87800000, len:509607936, virt:0x7ff707e00000,
> > > > > > > > socket_id:0, hugepage_sz:2097152, nchannel:0, nrank:0
> > > > > > > > dpdk_pool_create:550: ioctl(VFIO_IOMMU_MAP_DMA) pool
> > > > > > > > 'dpdk_mbuf_pool_socket0': Invalid argument (errno 22)
> > > > > > > > 0: dpdk_ipsec_process:1012: not enough DPDK crypto resources,
> > > > > > > > default to OpenSSL
> > > > > > > > 0: dpdk_lib_init:234: DPDK drivers found 2 ports...
> > > > > > > > _______ _ _ _____ ___
> > > > > > > > __/ __/ _ \ (_)__ | | / / _ \/ _ \
> > > > > > > > _/ _// // / / / _ \ | |/ / ___/ ___/
> > > > > > > > /_/ /____(_)_/\___/ |___/_/ /_/
> > > > > > > >
> > > > > > > > DBGvpp#
> > > > > > > >
> > > > > > > >
> > > > > > > >
> > > > > > > --
> > > > > > > Marco V
> > > > > > >
> > > > > > >
> > > > > > > SUSE LINUX GmbH | GF: Felix Imendörffer, Jane Smithard, Graham
> > > > > > > Norton
> > > > > > > HRB 21284 (AG Nürnberg) Maxfeldstr. 5, D-90409, Nürnberg
> > > > > >
> > > > > >
> > > > > >
> > > > > --
> > > > > Marco V
> > > > >
> > > > >
> > > > > SUSE LINUX GmbH | GF: Felix Imendörffer, Jane Smithard, Graham Norton
> > > > > HRB 21284 (AG Nürnberg) Maxfeldstr. 5, D-90409, Nürnberg
> > > >
> > > >
> > > >
> > > > --
> > > > Moon-Sang Lee, SW Engineer
> > > > Email: sang0...@gmail.com
> > > > Wisdom begins in wonder. *Socrates*
> > > >
> > > >
> > > >
> > > >
> > > >
> > >
> > >
> > >
> > --
> > Marco V
> >
> >
> > SUSE LINUX GmbH | GF: Felix Imendörffer, Jane Smithard, Graham Norton
> > HRB 21284 (AG Nürnberg) Maxfeldstr. 5, D-90409, Nürnberg
>
>
>
--
Marco V
SUSE LINUX GmbH | GF: Felix Imendörffer, Jane Smithard, Graham Norton
HRB 21284 (AG Nürnberg) Maxfeldstr. 5, D-90409, Nürnberg
