Damjan, In my case, we have a four-socket Skylake server filled with 2x25GbE NICs and encryption accelerators---trying to see how much IPsec throughput we can manage.
Recently saw a ~50% performance increase by changing from 2MB hugepages (8192 pages) to 1GB hugepages (16 pages) with VPP 18.04 on Ubuntu 16.04 (4.4.0 kernel). - Lee From: vpp-dev@lists.fd.io [mailto:vpp-dev@lists.fd.io] On Behalf Of Damjan Marion via Lists.Fd.Io Sent: Tuesday, October 23, 2018 4:05 PM To: Roberts, Lee A. <lee.robe...@hpe.com> Cc: vpp-dev@lists.fd.io Subject: Re: [vpp-dev] running VPP non-root broken I submitted patch.... Just for my curiosity what is the use case for allocating >=2 GB of wired memory? I thought nobody wants to do that so i did not pay attention. — Damjan On 23 Oct 2018, at 21:00, Lee Roberts <lee.robe...@hpe.com<mailto:lee.robe...@hpe.com>> wrote: In the following code, take a close look at line #61: <image001.jpg> Back in VPP 18.01.1, we had a similar problem---see VPP-1168 (https://jira.fd.io/browse/VPP-1168). One needs to be sure the left shift doesn’t lose high-order bits. You may need to explicitly cast “i” to be unsigned-64 before left-shifting it: uword pa = clib_pmalloc_get_pa (pm, (u8 *) va + ((u64)i << a->log2_page_sz)); - Lee Roberts From: vpp-dev@lists.fd.io<mailto:vpp-dev@lists.fd.io> [mailto:vpp-dev@lists.fd.io] On Behalf Of Sirshak Das Sent: Tuesday, October 23, 2018 12:45 PM To: dmar...@me.com<mailto:dmar...@me.com>; vpp-dev <vpp-dev@lists.fd.io<mailto:vpp-dev@lists.fd.io>> Cc: Honnappa Nagarahalli <honnappa.nagaraha...@arm.com<mailto:honnappa.nagaraha...@arm.com>>; Lijian Zhang (Arm Technology China) <lijian.zh...@arm.com<mailto:lijian.zh...@arm.com>>; khemendra kumar <khemendra.kuma...@gmail.com<mailto:khemendra.kuma...@gmail.com>>; Juraj Linkeš <juraj.lin...@pantheon.tech<mailto:juraj.lin...@pantheon.tech>> Subject: Re: [vpp-dev] running VPP non-root broken Hi Damjan, I am getting the following error as well I don’t know if its related to this issue: vlib_plugin_early_init:361: plugin path /home/sirdas/code/commita/vpp/build-root/install-vpp-native/vpp/lib/vpp_plugins load_one_plugin:117: Plugin disabled (default): abf_plugin.so load_one_plugin:117: Plugin disabled (default): acl_plugin.so load_one_plugin:117: Plugin disabled (default): avf_plugin.so load_one_plugin:117: Plugin disabled (default): cdp_plugin.so load_one_plugin:189: Loaded plugin: dpdk_plugin.so (Data Plane Development Kit (DPDK)) load_one_plugin:117: Plugin disabled (default): flowprobe_plugin.so load_one_plugin:117: Plugin disabled (default): gbp_plugin.so load_one_plugin:117: Plugin disabled (default): gtpu_plugin.so load_one_plugin:117: Plugin disabled (default): igmp_plugin.so load_one_plugin:117: Plugin disabled (default): ila_plugin.so load_one_plugin:117: Plugin disabled (default): ioam_plugin.so load_one_plugin:117: Plugin disabled (default): ixge_plugin.so load_one_plugin:117: Plugin disabled (default): l2e_plugin.so load_one_plugin:117: Plugin disabled (default): lacp_plugin.so load_one_plugin:117: Plugin disabled (default): lb_plugin.so load_one_plugin:117: Plugin disabled (default): mactime_plugin.so load_one_plugin:117: Plugin disabled (default): map_plugin.so load_one_plugin:117: Plugin disabled (default): memif_plugin.so load_one_plugin:117: Plugin disabled (default): nat_plugin.so load_one_plugin:117: Plugin disabled (default): nsh_plugin.so load_one_plugin:117: Plugin disabled (default): nsim_plugin.so load_one_plugin:117: Plugin disabled (default): perfmon_plugin.so load_one_plugin:117: Plugin disabled (default): pppoe_plugin.so load_one_plugin:117: Plugin disabled (default): srv6ad_plugin.so load_one_plugin:117: Plugin disabled (default): srv6am_plugin.so load_one_plugin:117: Plugin disabled (default): srv6as_plugin.so load_one_plugin:117: Plugin disabled (default): stn_plugin.so load_one_plugin:117: Plugin disabled (default): svs_plugin.so load_one_plugin:117: Plugin disabled (default): tlsmbedtls_plugin.so load_one_plugin:117: Plugin disabled (default): tlsopenssl_plugin.so load_one_plugin:117: Plugin disabled (default): unittest_plugin.so load_one_plugin:117: Plugin disabled (default): vmxnet3_plugin.so clib_elf_parse_file: open `linux-vdso.so.1': No such file or directory load_one_vat_plugin:67: Loaded plugin: lb_test_plugin.so load_one_vat_plugin:67: Loaded plugin: gtpu_test_plugin.so load_one_vat_plugin:67: Loaded plugin: memif_test_plugin.so load_one_vat_plugin:67: Loaded plugin: nsh_test_plugin.so load_one_vat_plugin:67: Loaded plugin: nsim_test_plugin.so load_one_vat_plugin:67: Loaded plugin: avf_test_plugin.so load_one_vat_plugin:67: Loaded plugin: vmxnet3_test_plugin.so load_one_vat_plugin:67: Loaded plugin: pppoe_test_plugin.so load_one_vat_plugin:67: Loaded plugin: mactime_test_plugin.so load_one_vat_plugin:67: Loaded plugin: lacp_test_plugin.so load_one_vat_plugin:67: Loaded plugin: dpdk_test_plugin.so load_one_vat_plugin:67: Loaded plugin: cdp_test_plugin.so load_one_vat_plugin:67: Loaded plugin: flowprobe_test_plugin.so load_one_vat_plugin:67: Loaded plugin: nat_test_plugin.so load_one_vat_plugin:67: Loaded plugin: stn_test_plugin.so load_one_vat_plugin:67: Loaded plugin: acl_test_plugin.so load_one_vat_plugin:67: Loaded plugin: ioam_test_plugin.so load_one_vat_plugin:67: Loaded plugin: map_test_plugin.so vnet_feature_arc_init:206: feature node 'acl-plugin-out-ip6-fa' not found (before 'ip6-dvr-reinject', arc 'ip6-output') vnet_feature_arc_init:206: feature node 'nat44-in2out-output' not found (before 'ip4-dvr-reinject', arc 'ip4-output') vnet_feature_arc_init:206: feature node 'acl-plugin-out-ip4-fa' not found (before 'ip4-dvr-reinject', arc 'ip4-output') vlib_physmem_shared_map_create: pmalloc_map_pages: failed to mmap 153 pages at 0xfffaa3c00000 fd 23 numa 0 flags 0x42031: Invalid argument dpdk_buffer_pool_create: failed to allocate mempool on socket 0 This is the startup.conf I am using: ip { heap-size 4G } unix { nodaemon interactive cli-listen localhost:5002 log /home/sirdas/var/log/vpp/vpp.log } ip6 { heap-size 4G hash-buckets 2000000 } heapsize 4G plugins { plugin default { disable } plugin dpdk_plugin.so { enable } } cpu { corelist-workers 18,20 main-core 17 } dpdk { dev 0004:01:00.1 dev 0004:01:00.2 no-multi-seg log-level debug dev default { num-rx-queues 2 num-rx-desc 2048 num-tx-desc 2048 } num-mbufs 128000 socket-mem 2048,2048 no-tx-checksum-offload } More info for debugging: Boot parameters: default_hugepagesz=1G hugepagesz=1G hugepages=32 hugepagesz=2M hugepages=2048 iommu.passthrough=1 isolcpus=16-45 nohz_full=16-45 rcu_nocbs=16-45 OS & Kernel: Ubuntu 18.04.1 LTS (GNU/Linux 4.15.0-38-generic aarch64) Meminfo: $ cat /proc/meminfo MemTotal: 98827048 kB MemFree: 2802448 kB MemAvailable: 2730464 kB Buffers: 44960 kB Cached: 477832 kB SwapCached: 0 kB Active: 368112 kB Inactive: 232532 kB Active(anon): 79504 kB Inactive(anon): 1312 kB Active(file): 288608 kB Inactive(file): 231220 kB Unevictable: 0 kB Mlocked: 0 kB SwapTotal: 96505852 kB SwapFree: 96505852 kB Dirty: 24 kB Writeback: 0 kB AnonPages: 78024 kB Mapped: 144960 kB Shmem: 2956 kB Slab: 207728 kB SReclaimable: 67780 kB SUnreclaim: 139948 kB KernelStack: 10464 kB PageTables: 2300 kB NFS_Unstable: 0 kB Bounce: 0 kB WritebackTmp: 0 kB CommitLimit: 98733456 kB Committed_AS: 940272 kB VmallocTotal: 135290290112 kB VmallocUsed: 0 kB VmallocChunk: 0 kB HardwareCorrupted: 0 kB AnonHugePages: 0 kB ShmemHugePages: 0 kB ShmemPmdMapped: 0 kB CmaTotal: 0 kB CmaFree: 0 kB HugePages_Total: 86 HugePages_Free: 86 HugePages_Rsvd: 0 HugePages_Surp: 0 Hugepagesize: 1048576 kB Let me know if I am doing anything wrong: This is VPP (master branch) with HEAD at: commit 68b4da67deb2e8ca224bb5abaeb9dbc7ae8e378c (HEAD -> master, origin/master, origin/HEAD) Author: Damjan Marion <damar...@cisco.com<mailto:damar...@cisco.com>> Date: Sun Sep 30 18:26:20 2018 +0200 Numa-aware, growable physical memory allocator (pmalloc) Change-Id: Ic4c46bc733afae8bf0d8146623ed15633928de30 Signed-off-by: Damjan Marion <damar...@cisco.com<mailto:damar...@cisco.com>> Thank you Sirshak Das From: vpp-dev@lists.fd.io<mailto:vpp-dev@lists.fd.io> <vpp-dev@lists.fd.io<mailto:vpp-dev@lists.fd.io>> On Behalf Of Damjan Marion via Lists.Fd.Io Sent: Tuesday, October 23, 2018 11:40 AM To: vpp-dev <vpp-dev@lists.fd.io<mailto:vpp-dev@lists.fd.io>> Cc: vpp-dev@lists.fd.io<mailto:vpp-dev@lists.fd.io> Subject: [vpp-dev] running VPP non-root broken Folks, Looks like my big physmem patch breaks non-root operation of VPP, working on it and It will take a bit of time so as a workaround "make test" can be run with sudo. Let me know if any issues, and I will revert, but would like to avoid that due to the size of patch. -- Damjan IMPORTANT NOTICE: The contents of this email and any attachments are confidential and may also be privileged. If you are not the intended recipient, please notify the sender immediately and do not disclose the contents to any other person, use it for any purpose, or store or copy the information in any medium. Thank you. -=-=-=-=-=-=-=-=-=-=-=- Links: You receive all messages sent to this group. View/Reply Online (#10929): https://lists.fd.io/g/vpp-dev/message/10929 Mute This Topic: https://lists.fd.io/mt/27570325/675642 Group Owner: vpp-dev+ow...@lists.fd.io<mailto:vpp-dev+ow...@lists.fd.io> Unsubscribe: https://lists.fd.io/g/vpp-dev/unsub [dmar...@me.com<mailto:dmar...@me.com>] -=-=-=-=-=-=-=-=-=-=-=-
-=-=-=-=-=-=-=-=-=-=-=- Links: You receive all messages sent to this group. View/Reply Online (#10935): https://lists.fd.io/g/vpp-dev/message/10935 Mute This Topic: https://lists.fd.io/mt/27570325/21656 Group Owner: vpp-dev+ow...@lists.fd.io Unsubscribe: https://lists.fd.io/g/vpp-dev/unsub [arch...@mail-archive.com] -=-=-=-=-=-=-=-=-=-=-=-
