We merged patch which should fix things with 1G hugepages but I was not able to
test it on arm, so please try...
--
Damjan
> On 24 Oct 2018, at 05:28, Sirshak Das <sirshak....@arm.com> wrote:
>
> 2M works but 1G still fails.
>
> I toned down the dpdk resource allocation to default:
> dpdk
> {
> dev 0004:01:00.1
> dev 0004:01:00.2
> no-multi-seg
> log-level debug
> dev default
> {
> num-rx-queues 1
> # num-tx-queues 4
> num-rx-desc 2048
> num-tx-desc 2048
> }
> # num-mbufs 128000
> # socket-mem 2048,2048
> no-tx-checksum-offload
>
> But here is the problem (for 16G of Hugepage memory):
> With:
> 2MB (nr_hugepages: 8192)
> GRUB_CMDLINE_LINUX="default_hugepagesz=2M hugepagesz=1G hugepages=16
> hugepagesz=2M hugepages=8192 iommu.passthrough=1 isolcpus=16-45
> nohz_full=16-45 rcu_nocbs=16-45"
> vs 1GB (nr_hugepages: 16)
> GRUB_CMDLINE_LINUX="default_hugepagesz=1G hugepagesz=1G hugepages=16
> hugepagesz=2M hugepages=8192 iommu.passthrough=1 isolcpus=16-45
> nohz_full=16-45 rcu_nocbs=16-45"
>
> I am getting a performance improvement of 49% when I use 1G hugepages
> compared to 2MB.
> I am not an expert on hugepages to pinpoint the exact reason but it will
> surely help if you can fix the 1G hugepage issue.
>
> Thank you
> Sirshak Das
> From: Damjan Marion <dmar...@me.com>
> Sent: Tuesday, October 23, 2018 3:43 PM
> To: Sirshak Das <sirshak....@arm.com>
> Cc: vpp-dev <vpp-dev@lists.fd.io>; Honnappa Nagarahalli
> <honnappa.nagaraha...@arm.com>; Lijian Zhang (Arm Technology China)
> <lijian.zh...@arm.com>; khemendra kumar <khemendra.kuma...@gmail.com>; Juraj
> Linkeš <juraj.lin...@pantheon.tech>
> Subject: Re: [vpp-dev] running VPP non-root broken
>
>
> OMG, you are good in wasting memory. 1G pages, 2G per socket given to dpdk to
> hang empty :)
> 128K buffers....
>
> For a start can you switch default page size to 2M. newer x86 kernels ignore
> it but maybe it behaves
> differently on aarch64...
>
> In the meantime I will fix few coverity issues...
>
> --
> Damjan
>
>
> On 23 Oct 2018, at 20:45, Sirshak Das <sirshak....@arm.com
> <mailto:sirshak....@arm.com>> wrote:
>
> Hi Damjan,
>
> I am getting the following error as well I don’t know if its related to this
> issue:
> vlib_plugin_early_init:361: plugin path
> /home/sirdas/code/commita/vpp/build-root/install-vpp-native/vpp/lib/vpp_plugins
> load_one_plugin:117: Plugin disabled (default): abf_plugin.so
> load_one_plugin:117: Plugin disabled (default): acl_plugin.so
> load_one_plugin:117: Plugin disabled (default): avf_plugin.so
> load_one_plugin:117: Plugin disabled (default): cdp_plugin.so
> load_one_plugin:189: Loaded plugin: dpdk_plugin.so (Data Plane Development
> Kit (DPDK))
> load_one_plugin:117: Plugin disabled (default): flowprobe_plugin.so
> load_one_plugin:117: Plugin disabled (default): gbp_plugin.so
> load_one_plugin:117: Plugin disabled (default): gtpu_plugin.so
> load_one_plugin:117: Plugin disabled (default): igmp_plugin.so
> load_one_plugin:117: Plugin disabled (default): ila_plugin.so
> load_one_plugin:117: Plugin disabled (default): ioam_plugin.so
> load_one_plugin:117: Plugin disabled (default): ixge_plugin.so
> load_one_plugin:117: Plugin disabled (default): l2e_plugin.so
> load_one_plugin:117: Plugin disabled (default): lacp_plugin.so
> load_one_plugin:117: Plugin disabled (default): lb_plugin.so
> load_one_plugin:117: Plugin disabled (default): mactime_plugin.so
> load_one_plugin:117: Plugin disabled (default): map_plugin.so
> load_one_plugin:117: Plugin disabled (default): memif_plugin.so
> load_one_plugin:117: Plugin disabled (default): nat_plugin.so
> load_one_plugin:117: Plugin disabled (default): nsh_plugin.so
> load_one_plugin:117: Plugin disabled (default): nsim_plugin.so
> load_one_plugin:117: Plugin disabled (default): perfmon_plugin.so
> load_one_plugin:117: Plugin disabled (default): pppoe_plugin.so
> load_one_plugin:117: Plugin disabled (default): srv6ad_plugin.so
> load_one_plugin:117: Plugin disabled (default): srv6am_plugin.so
> load_one_plugin:117: Plugin disabled (default): srv6as_plugin.so
> load_one_plugin:117: Plugin disabled (default): stn_plugin.so
> load_one_plugin:117: Plugin disabled (default): svs_plugin.so
> load_one_plugin:117: Plugin disabled (default): tlsmbedtls_plugin.so
> load_one_plugin:117: Plugin disabled (default): tlsopenssl_plugin.so
> load_one_plugin:117: Plugin disabled (default): unittest_plugin.so
> load_one_plugin:117: Plugin disabled (default): vmxnet3_plugin.so
> clib_elf_parse_file: open `linux-vdso.so.1': No such file or directory
> load_one_vat_plugin:67: Loaded plugin: lb_test_plugin.so
> load_one_vat_plugin:67: Loaded plugin: gtpu_test_plugin.so
> load_one_vat_plugin:67: Loaded plugin: memif_test_plugin.so
> load_one_vat_plugin:67: Loaded plugin: nsh_test_plugin.so
> load_one_vat_plugin:67: Loaded plugin: nsim_test_plugin.so
> load_one_vat_plugin:67: Loaded plugin: avf_test_plugin.so
> load_one_vat_plugin:67: Loaded plugin: vmxnet3_test_plugin.so
> load_one_vat_plugin:67: Loaded plugin: pppoe_test_plugin.so
> load_one_vat_plugin:67: Loaded plugin: mactime_test_plugin.so
> load_one_vat_plugin:67: Loaded plugin: lacp_test_plugin.so
> load_one_vat_plugin:67: Loaded plugin: dpdk_test_plugin.so
> load_one_vat_plugin:67: Loaded plugin: cdp_test_plugin.so
> load_one_vat_plugin:67: Loaded plugin: flowprobe_test_plugin.so
> load_one_vat_plugin:67: Loaded plugin: nat_test_plugin.so
> load_one_vat_plugin:67: Loaded plugin: stn_test_plugin.so
> load_one_vat_plugin:67: Loaded plugin: acl_test_plugin.so
> load_one_vat_plugin:67: Loaded plugin: ioam_test_plugin.so
> load_one_vat_plugin:67: Loaded plugin: map_test_plugin.so
> vnet_feature_arc_init:206: feature node 'acl-plugin-out-ip6-fa' not found
> (before 'ip6-dvr-reinject', arc 'ip6-output')
> vnet_feature_arc_init:206: feature node 'nat44-in2out-output' not found
> (before 'ip4-dvr-reinject', arc 'ip4-output')
> vnet_feature_arc_init:206: feature node 'acl-plugin-out-ip4-fa' not found
> (before 'ip4-dvr-reinject', arc 'ip4-output')
> vlib_physmem_shared_map_create: pmalloc_map_pages: failed to mmap 153 pages
> at 0xfffaa3c00000 fd 23 numa 0 flags 0x42031: Invalid argument
>
> dpdk_buffer_pool_create: failed to allocate mempool on socket 0
>
>
> This is the startup.conf I am using:
>
> ip
> {
> heap-size 4G
> }
> unix
> {
> nodaemon
> interactive
> cli-listen localhost:5002
> log /home/sirdas/var/log/vpp/vpp.log
> }
> ip6
> {
> heap-size 4G
> hash-buckets 2000000
> }
> heapsize 4G
> plugins
> {
> plugin default
> {
> disable
> }
> plugin dpdk_plugin.so
> {
> enable
> }
> }
> cpu
> {
> corelist-workers 18,20
> main-core 17
> }
> dpdk
> {
> dev 0004:01:00.1
> dev 0004:01:00.2
> no-multi-seg
> log-level debug
> dev default
> {
> num-rx-queues 2
> num-rx-desc 2048
> num-tx-desc 2048
> }
> num-mbufs 128000
> socket-mem 2048,2048
> no-tx-checksum-offload
> }
>
> More info for debugging:
> Boot parameters:
> default_hugepagesz=1G hugepagesz=1G hugepages=32 hugepagesz=2M hugepages=2048
> iommu.passthrough=1 isolcpus=16-45 nohz_full=16-45 rcu_nocbs=16-45
>
> OS & Kernel:
> Ubuntu 18.04.1 LTS (GNU/Linux 4.15.0-38-generic aarch64)
>
> Meminfo:
> $ cat /proc/meminfo
> MemTotal: 98827048 kB
> MemFree: 2802448 kB
> MemAvailable: 2730464 kB
> Buffers: 44960 kB
> Cached: 477832 kB
> SwapCached: 0 kB
> Active: 368112 kB
> Inactive: 232532 kB
> Active(anon): 79504 kB
> Inactive(anon): 1312 kB
> Active(file): 288608 kB
> Inactive(file): 231220 kB
> Unevictable: 0 kB
> Mlocked: 0 kB
> SwapTotal: 96505852 kB
> SwapFree: 96505852 kB
> Dirty: 24 kB
> Writeback: 0 kB
> AnonPages: 78024 kB
> Mapped: 144960 kB
> Shmem: 2956 kB
> Slab: 207728 kB
> SReclaimable: 67780 kB
> SUnreclaim: 139948 kB
> KernelStack: 10464 kB
> PageTables: 2300 kB
> NFS_Unstable: 0 kB
> Bounce: 0 kB
> WritebackTmp: 0 kB
> CommitLimit: 98733456 kB
> Committed_AS: 940272 kB
> VmallocTotal: 135290290112 <tel:135290290112> kB
> VmallocUsed: 0 kB
> VmallocChunk: 0 kB
> HardwareCorrupted: 0 kB
> AnonHugePages: 0 kB
> ShmemHugePages: 0 kB
> ShmemPmdMapped: 0 kB
> CmaTotal: 0 kB
> CmaFree: 0 kB
> HugePages_Total: 86
> HugePages_Free: 86
> HugePages_Rsvd: 0
> HugePages_Surp: 0
> Hugepagesize: 1048576 <tel:1048576> kB
>
> Let me know if I am doing anything wrong:
>
> This is VPP (master branch) with HEAD at:
>
> commit 68b4da67deb2e8ca224bb5abaeb9dbc7ae8e378c (HEAD -> master,
> origin/master, origin/HEAD)
> Author: Damjan Marion <damar...@cisco.com <mailto:damar...@cisco.com>>
> Date: Sun Sep 30 18:26:20 2018 +0200
>
> Numa-aware, growable physical memory allocator (pmalloc)
>
> Change-Id: Ic4c46bc733afae8bf0d8146623ed15633928de30
> Signed-off-by: Damjan Marion <damar...@cisco.com
> <mailto:damar...@cisco.com>>
>
>
> Thank you
> Sirshak Das
> From: vpp-dev@lists.fd.io <mailto:vpp-dev@lists.fd.io> <vpp-dev@lists.fd.io
> <mailto:vpp-dev@lists.fd.io>> On Behalf Of Damjan Marion via Lists.Fd.Io
> Sent: Tuesday, October 23, 2018 11:40 AM
> To: vpp-dev <vpp-dev@lists.fd.io <mailto:vpp-dev@lists.fd.io>>
> Cc: vpp-dev@lists.fd.io <mailto:vpp-dev@lists.fd.io>
> Subject: [vpp-dev] running VPP non-root broken
>
>
> Folks,
>
> Looks like my big physmem patch breaks non-root operation of VPP, working on
> it
> and It will take a bit of time so as a workaround "make test" can be run with
> sudo.
>
> Let me know if any issues, and I will revert, but would like to avoid that
> due to the size of patch.
>
> --
> Damjan
>
> IMPORTANT NOTICE: The contents of this email and any attachments are
> confidential and may also be privileged. If you are not the intended
> recipient, please notify the sender immediately and do not disclose the
> contents to any other person, use it for any purpose, or store or copy the
> information in any medium. Thank you.
>
> IMPORTANT NOTICE: The contents of this email and any attachments are
> confidential and may also be privileged. If you are not the intended
> recipient, please notify the sender immediately and do not disclose the
> contents to any other person, use it for any purpose, or store or copy the
> information in any medium. Thank you.
-=-=-=-=-=-=-=-=-=-=-=-
Links: You receive all messages sent to this group.
View/Reply Online (#10957): https://lists.fd.io/g/vpp-dev/message/10957
Mute This Topic: https://lists.fd.io/mt/27570325/21656
Group Owner: vpp-dev+ow...@lists.fd.io
Unsubscribe: https://lists.fd.io/g/vpp-dev/unsub [arch...@mail-archive.com]
-=-=-=-=-=-=-=-=-=-=-=-
