Dear Filip,
Thank you very much for your prompt reply!
I have attached the startup.conf files for the case with 2 workers
(startup.conf-mc1wc02) and for the case when I used only the main core
(startup.conf-mc0). In both cases, 4 CPU cores (0-3) were enabled and
cores 0-2 were excluded from the Linux scheduler using the "maxcpus=4"
and "isolcpus=0-2" kernel command line parameters, respectively.
I am new to FD.io VPP. Could you please advise me if there are pre-built
debug packages available for Debian 10, and if yes, where can I find them?
If I need to compile them myself, could you please give me a pointer,
how I can do it?
I am currently using servers in NICT StarBED, Japan. This is a test-bed
environment, and I can download packages only using a http or ftp proxy.
(Or I can download them on my Windows laptop and upload them through a
gateway.)
Thank you very much in advance!
Best regards,
Gábor
On 11/9/2022 4:04 PM, filvarga wrote:
Hi Gabor,
I will look into it and get back to you. Meanwhile could you run the
same test with a debug build and post the results ? Maybe even core
dump. Also please post your startup.conf file
Best regards,
Filip Varga
st 9. 11. 2022 o 7:50 Gabor LENCSE <len...@hit.bme.hu> napísal(a):
Dear VPP Developers,
I am a researcher and I would like to benchmark the performance of
the stateful NAT64 implementation of FD.io VPP.
Unfortunately, VPP crashed with segmentation fault.
Some details:
I used two Dell PowerEdge R430 servers as the Tester and the DUT
(Device Under Test), two 10GbE interfaces of which were
interconnected by direct cables. On the DUT, I used Debian Linux
10.13 with 4.19.0-20-amd64 kernel and the version of FD.io VPP was
22.06. The following packages were installed: libvppinfra, vpp,
vpp-plugin-core, vpp-plugin-dpdk.
I used the following commands to set up Stateful NAT64:
root@p109:~/DUT-settings# cat set-vpp
vppctl set interface state TenGigabitEthernet5/0/0 up
vppctl set interface state TenGigabitEthernet5/0/1 up
vppctl set interface ip address TenGigabitEthernet5/0/0 2001:2::1/64
vppctl set interface ip address TenGigabitEthernet5/0/1
198.19.0.1/24 <http://198.19.0.1/24>
vppctl ip route add 2001:2::/64 via 2001:2::1 TenGigabitEthernet5/0/0
vppctl ip route add 198.19.0.0/24 <http://198.19.0.0/24> via
198.19.0.1 TenGigabitEthernet5/0/1
vppctl set ip neighbor static TenGigabitEthernet5/0/0 2001:2::2
a0:36:9f:74:73:64
vppctl set ip neighbor static TenGigabitEthernet5/0/1 198.19.0.2
a0:36:9f:74:73:66
vppctl set interface nat64 in TenGigabitEthernet5/0/0
vppctl set interface nat64 out TenGigabitEthernet5/0/1
vppctl nat64 add prefix 64:ff9b::/96
vppctl nat64 add pool address 198.19.0.1
As for VPP, first I used two workers, but then I also tried
without workers, using only the main core. Unfortunately, VPP
crashed in both cases, but with somewhat different messages in the
syslog. (Previously I tested both setups with IPv6 packet
forwarding and they worked with an excellent performance.)
The error messages in the syslog when I used two workers:
Nov 7 16:32:02 p109 vnet[2479]: received signal SIGSEGV, PC
0x7fa86f138168, faulting address 0x4f8
Nov 7 16:32:02 p109 vnet[2479]: #0 0x00007fa8b2158137 0x7fa8b2158137
Nov 7 16:32:02 p109 vnet[2479]: #1 0x00007fa8b2086730 0x7fa8b2086730
Nov 7 16:32:02 p109 vnet[2479]: #2 0x00007fa86f138168 0x7fa86f138168
Nov 7 16:32:02 p109 vnet[2479]: #3 0x00007fa86f11d228 0x7fa86f11d228
Nov 7 16:32:02 p109 vnet[2479]: #4 0x00007fa8b20fbe62 0x7fa8b20fbe62
Nov 7 16:32:02 p109 vnet[2479]: #5 0x00007fa8b20fda4f
vlib_worker_loop + 0x5ff
Nov 7 16:32:02 p109 vnet[2479]: #6 0x00007fa8b2135e79
vlib_worker_thread_fn + 0xa9
Nov 7 16:32:02 p109 vnet[2479]: #7 0x00007fa8b2135290
vlib_worker_thread_bootstrap_fn + 0x50
Nov 7 16:32:02 p109 vnet[2479]: #8 0x00007fa8b207bfa3
start_thread + 0xf3
Nov 7 16:32:02 p109 vnet[2479]: #9 0x00007fa8b1d75eff clone + 0x3f
Nov 7 16:32:02 p109 systemd[1]: vpp.service: Main process exited,
code=killed, status=6/ABRT
The error messages in the syslog when I used only the main core:
Nov 7 16:48:57 p109 vnet[2606]: received signal SIGSEGV, PC
0x7fbe1d24a168, faulting address 0x1a8
Nov 7 16:48:57 p109 vnet[2606]: #0 0x00007fbe6026a137 0x7fbe6026a137
Nov 7 16:48:57 p109 vnet[2606]: #1 0x00007fbe60198730 0x7fbe60198730
Nov 7 16:48:57 p109 vnet[2606]: #2 0x00007fbe1d24a168 0x7fbe1d24a168
Nov 7 16:48:57 p109 vnet[2606]: #3 0x00007fbe1d22f228 0x7fbe1d22f228
Nov 7 16:48:57 p109 vnet[2606]: #4 0x00007fbe6020de62 0x7fbe6020de62
Nov 7 16:48:57 p109 vnet[2606]: #5 0x00007fbe602127d1 vlib_main
+ 0xd41
Nov 7 16:48:57 p109 vnet[2606]: #6 0x00007fbe6026906a 0x7fbe6026906a
Nov 7 16:48:57 p109 vnet[2606]: #7 0x00007fbe60169964 0x7fbe60169964
Nov 7 16:48:57 p109 systemd[1]: vpp.service: Main process exited,
code=killed, status=6/ABRT
As for the first time I started with quite high load, I suspected
that I exhausted some sort of resources, so I tried with much
lower load, but the same thing happened even when I sent only a
single packet.
I used siitperf as Tester: https://github.com/lencsegabor/siitperf
And I followed this methodology:
https://datatracker.ietf.org/doc/html/draft-ietf-bmwg-benchmarking-stateful
Previously my tests were successful with the following stateful
NAT64 implementations:
- Jool
- tayga+iptables
- OpenBSD PF
Could you please help me why VPP crashes, and how I could make it
work?
Thank you very much for your help in advance!
Best regards,
Gábor Lencse
unix {
nodaemon
log /var/log/vpp/vpp.log
full-coredump
cli-listen /run/vpp/cli.sock
gid vpp
## run vpp in the interactive mode
# interactive
## do not use colors in terminal output
# nocolor
## do not display banner
# nobanner
}
api-trace {
## This stanza controls binary API tracing. Unless there is a very strong
reason,
## please leave this feature enabled.
on
## Additional parameters:
##
## To set the number of binary API trace records in the circular buffer,
configure nitems
##
## nitems <nnn>
##
## To save the api message table decode tables, configure a filename. Results
in /tmp/<filename>
## Very handy for understanding api message changes between versions,
identifying missing
## plugins, and so forth.
##
## save-api-table <filename>
}
api-segment {
gid vpp
}
socksvr {
default
}
# memory {
## Set the main heap size, default is 1G
# main-heap-size 2G
## Set the main heap page size. Default page size is OS default page
## which is in most cases 4K. if different page size is specified VPP
## will try to allocate main heap by using specified page size.
## special keyword 'default-hugepage' will use system default hugepage
## size
# main-heap-page-size 1G
## Set the default huge page size.
# default-hugepage-size 1G
#}
cpu {
## In the VPP there is one main thread and optionally the user can
create worker(s)
## The main thread and worker thread(s) can be pinned to CPU core(s)
manually or automatically
## Manual pinning of thread(s) to CPU core(s)
## Set logical CPU core where main thread runs, if main core is not set
## VPP will use core 1 if available
main-core 1
## Set logical CPU core(s) where worker threads are running
corelist-workers 0,2
## Automatic pinning of thread(s) to CPU core(s)
## Sets number of CPU core(s) to be skipped (1 ... N-1)
## Skipped CPU core(s) are not used for pinning main thread and working
thread(s).
## The main thread is automatically pinned to the first available CPU
core and worker(s)
## are pinned to next free CPU core(s) after core assigned to main
thread
# skip-cores 4
## Specify a number of workers to be created
## Workers are pinned to N consecutive CPU cores while skipping
"skip-cores" CPU core(s)
## and main thread's CPU core
# workers 2
## Set scheduling policy and priority of main and worker threads
## Scheduling policy options are: other (SCHED_OTHER), batch
(SCHED_BATCH)
## idle (SCHED_IDLE), fifo (SCHED_FIFO), rr (SCHED_RR)
# scheduler-policy fifo
## Scheduling priority is used only for "real-time policies (fifo and
rr),
## and has to be in the range of priorities supported for a particular
policy
# scheduler-priority 50
}
# buffers {
## Increase number of buffers allocated, needed only in scenarios with
## large number of interfaces and worker threads. Value is per numa
node.
## Default is 16384 (8192 if running unpriviledged)
# buffers-per-numa 128000
## Size of buffer data area
## Default is 2048
# default data-size 2048
## Size of the memory pages allocated for buffer data
## Default will try 'default-hugepage' then 'default'
## you can also pass a size in K/M/G e.g. '8M'
# page-size default-hugepage
# }
# dpdk {
## Change default settings for all interfaces
# dev default {
## Number of receive queues, enables RSS
## Default is 1
# num-rx-queues 3
## Number of transmit queues, Default is equal
## to number of worker threads or 1 if no workers treads
# num-tx-queues 3
## Number of descriptors in transmit and receive rings
## increasing or reducing number can impact performance
## Default is 1024 for both rx and tx
# num-rx-desc 512
# num-tx-desc 512
## TCP Segment Offload
## Default is off
## To enable TSO, 'enable-tcp-udp-checksum' must be set
# tso on
## Devargs
## device specific init args
## Default is NULL
# devargs safe-mode-support=1,pipeline-mode-support=1
## rss-queues
## set valid rss steering queues
# rss-queues 0,2,5-7
# }
## Whitelist specific interface by specifying PCI address
# dev 0000:02:00.0
## Blacklist specific device type by specifying PCI vendor:device
## Whitelist entries take precedence
# blacklist 8086:10fb
## Set interface name
# dev 0000:02:00.1 {
# name eth0
# }
## Whitelist specific interface by specifying PCI address and in
## addition specify custom parameters for this interface
# dev 0000:02:00.1 {
# num-rx-queues 2
# }
## Change UIO driver used by VPP, Options are: igb_uio, vfio-pci,
## uio_pci_generic or auto (default)
# uio-driver vfio-pci
## Disable multi-segment buffers, improves performance but
## disables Jumbo MTU support
# no-multi-seg
## Change hugepages allocation per-socket, needed only if there is need
for
## larger number of mbufs. Default is 256M on each detected CPU socket
# socket-mem 2048,2048
## Disables UDP / TCP TX checksum offload. Typically needed for use
## faster vector PMDs (together with no-multi-seg)
# no-tx-checksum-offload
## Enable UDP / TCP TX checksum offload
## This is the reversed option of 'no-tx-checksum-offload'
# enable-tcp-udp-checksum
## Enable/Disable AVX-512 vPMDs
# max-simd-bitwidth <256|512>
# }
## node variant defaults
#node {
## specify the preferred default variant
# default { variant avx512 }
## specify the preferred variant, for a given node
# ip4-rewrite { variant avx2 }
#}
# plugins {
## Adjusting the plugin path depending on where the VPP plugins are
# path /ws/vpp/build-root/install-vpp-native/vpp/lib/vpp_plugins
## Add additional directory to the plugin path
# add-path /tmp/vpp_plugins
## Disable all plugins by default and then selectively enable specific
plugins
# plugin default { disable }
# plugin dpdk_plugin.so { enable }
# plugin acl_plugin.so { enable }
## Enable all plugins by default and then selectively disable specific
plugins
# plugin dpdk_plugin.so { disable }
# plugin acl_plugin.so { disable }
# }
## Statistics Segment
# statseg {
# socket-name <filename>, name of the stats segment socket
# defaults to /run/vpp/stats.sock
# size <nnn>[KMG], size of the stats segment, defaults to 32mb
# page-size <nnn>, page size, ie. 2m, defaults to 4k
# per-node-counters on | off, defaults to none
# update-interval <f64-seconds>, sets the segment scrape / update interval
# }
## L2 FIB
# l2fib {
## l2fib hash table size.
# table-size 512M
## l2fib hash table number of buckets. Must be power of 2.
# num-buckets 524288
# }
## ipsec
# {
# ip4 {
## ipsec for ipv4 tunnel lookup hash number of buckets.
# num-buckets 524288
# }
# ip6 {
## ipsec for ipv6 tunnel lookup hash number of buckets.
# num-buckets 524288
# }
# }
# logging {
## set default logging level for logging buffer
## logging levels: emerg, alert,crit, error, warn, notice, info, debug,
disabled
# default-log-level debug
## set default logging level for syslog or stderr output
# default-syslog-log-level info
## Set per-class configuration
# class dpdk/cryptodev { rate-limit 100 level debug syslog-level error }
# }
unix {
nodaemon
log /var/log/vpp/vpp.log
full-coredump
cli-listen /run/vpp/cli.sock
gid vpp
## run vpp in the interactive mode
# interactive
## do not use colors in terminal output
# nocolor
## do not display banner
# nobanner
}
api-trace {
## This stanza controls binary API tracing. Unless there is a very strong
reason,
## please leave this feature enabled.
on
## Additional parameters:
##
## To set the number of binary API trace records in the circular buffer,
configure nitems
##
## nitems <nnn>
##
## To save the api message table decode tables, configure a filename. Results
in /tmp/<filename>
## Very handy for understanding api message changes between versions,
identifying missing
## plugins, and so forth.
##
## save-api-table <filename>
}
api-segment {
gid vpp
}
socksvr {
default
}
# memory {
## Set the main heap size, default is 1G
# main-heap-size 2G
## Set the main heap page size. Default page size is OS default page
## which is in most cases 4K. if different page size is specified VPP
## will try to allocate main heap by using specified page size.
## special keyword 'default-hugepage' will use system default hugepage
## size
# main-heap-page-size 1G
## Set the default huge page size.
# default-hugepage-size 1G
#}
cpu {
## In the VPP there is one main thread and optionally the user can
create worker(s)
## The main thread and worker thread(s) can be pinned to CPU core(s)
manually or automatically
## Manual pinning of thread(s) to CPU core(s)
## Set logical CPU core where main thread runs, if main core is not set
## VPP will use core 1 if available
main-core 0
## Set logical CPU core(s) where worker threads are running
# corelist-workers 2-3,18-19
## Automatic pinning of thread(s) to CPU core(s)
## Sets number of CPU core(s) to be skipped (1 ... N-1)
## Skipped CPU core(s) are not used for pinning main thread and working
thread(s).
## The main thread is automatically pinned to the first available CPU
core and worker(s)
## are pinned to next free CPU core(s) after core assigned to main
thread
# skip-cores 4
## Specify a number of workers to be created
## Workers are pinned to N consecutive CPU cores while skipping
"skip-cores" CPU core(s)
## and main thread's CPU core
# workers 2
## Set scheduling policy and priority of main and worker threads
## Scheduling policy options are: other (SCHED_OTHER), batch
(SCHED_BATCH)
## idle (SCHED_IDLE), fifo (SCHED_FIFO), rr (SCHED_RR)
# scheduler-policy fifo
## Scheduling priority is used only for "real-time policies (fifo and
rr),
## and has to be in the range of priorities supported for a particular
policy
# scheduler-priority 50
}
# buffers {
## Increase number of buffers allocated, needed only in scenarios with
## large number of interfaces and worker threads. Value is per numa
node.
## Default is 16384 (8192 if running unpriviledged)
# buffers-per-numa 128000
## Size of buffer data area
## Default is 2048
# default data-size 2048
## Size of the memory pages allocated for buffer data
## Default will try 'default-hugepage' then 'default'
## you can also pass a size in K/M/G e.g. '8M'
# page-size default-hugepage
# }
# dpdk {
## Change default settings for all interfaces
# dev default {
## Number of receive queues, enables RSS
## Default is 1
# num-rx-queues 3
## Number of transmit queues, Default is equal
## to number of worker threads or 1 if no workers treads
# num-tx-queues 3
## Number of descriptors in transmit and receive rings
## increasing or reducing number can impact performance
## Default is 1024 for both rx and tx
# num-rx-desc 512
# num-tx-desc 512
## TCP Segment Offload
## Default is off
## To enable TSO, 'enable-tcp-udp-checksum' must be set
# tso on
## Devargs
## device specific init args
## Default is NULL
# devargs safe-mode-support=1,pipeline-mode-support=1
## rss-queues
## set valid rss steering queues
# rss-queues 0,2,5-7
# }
## Whitelist specific interface by specifying PCI address
# dev 0000:02:00.0
## Blacklist specific device type by specifying PCI vendor:device
## Whitelist entries take precedence
# blacklist 8086:10fb
## Set interface name
# dev 0000:02:00.1 {
# name eth0
# }
## Whitelist specific interface by specifying PCI address and in
## addition specify custom parameters for this interface
# dev 0000:02:00.1 {
# num-rx-queues 2
# }
## Change UIO driver used by VPP, Options are: igb_uio, vfio-pci,
## uio_pci_generic or auto (default)
# uio-driver vfio-pci
## Disable multi-segment buffers, improves performance but
## disables Jumbo MTU support
# no-multi-seg
## Change hugepages allocation per-socket, needed only if there is need
for
## larger number of mbufs. Default is 256M on each detected CPU socket
# socket-mem 2048,2048
## Disables UDP / TCP TX checksum offload. Typically needed for use
## faster vector PMDs (together with no-multi-seg)
# no-tx-checksum-offload
## Enable UDP / TCP TX checksum offload
## This is the reversed option of 'no-tx-checksum-offload'
# enable-tcp-udp-checksum
## Enable/Disable AVX-512 vPMDs
# max-simd-bitwidth <256|512>
# }
## node variant defaults
#node {
## specify the preferred default variant
# default { variant avx512 }
## specify the preferred variant, for a given node
# ip4-rewrite { variant avx2 }
#}
# plugins {
## Adjusting the plugin path depending on where the VPP plugins are
# path /ws/vpp/build-root/install-vpp-native/vpp/lib/vpp_plugins
## Add additional directory to the plugin path
# add-path /tmp/vpp_plugins
## Disable all plugins by default and then selectively enable specific
plugins
# plugin default { disable }
# plugin dpdk_plugin.so { enable }
# plugin acl_plugin.so { enable }
## Enable all plugins by default and then selectively disable specific
plugins
# plugin dpdk_plugin.so { disable }
# plugin acl_plugin.so { disable }
# }
## Statistics Segment
# statseg {
# socket-name <filename>, name of the stats segment socket
# defaults to /run/vpp/stats.sock
# size <nnn>[KMG], size of the stats segment, defaults to 32mb
# page-size <nnn>, page size, ie. 2m, defaults to 4k
# per-node-counters on | off, defaults to none
# update-interval <f64-seconds>, sets the segment scrape / update interval
# }
## L2 FIB
# l2fib {
## l2fib hash table size.
# table-size 512M
## l2fib hash table number of buckets. Must be power of 2.
# num-buckets 524288
# }
## ipsec
# {
# ip4 {
## ipsec for ipv4 tunnel lookup hash number of buckets.
# num-buckets 524288
# }
# ip6 {
## ipsec for ipv6 tunnel lookup hash number of buckets.
# num-buckets 524288
# }
# }
# logging {
## set default logging level for logging buffer
## logging levels: emerg, alert,crit, error, warn, notice, info, debug,
disabled
# default-log-level debug
## set default logging level for syslog or stderr output
# default-syslog-log-level info
## Set per-class configuration
# class dpdk/cryptodev { rate-limit 100 level debug syslog-level error }
# }
-=-=-=-=-=-=-=-=-=-=-=-
Links: You receive all messages sent to this group.
View/Reply Online (#22155): https://lists.fd.io/g/vpp-dev/message/22155
Mute This Topic: https://lists.fd.io/mt/94908130/21656
Group Owner: vpp-dev+ow...@lists.fd.io
Unsubscribe: https://lists.fd.io/g/vpp-dev/leave/1480452/21656/631435203/xyzzy
[arch...@mail-archive.com]
-=-=-=-=-=-=-=-=-=-=-=-
