Dear VPP Developers,
I am a researcher and I would like to benchmark the performance of the
stateful NAT64 implementation of FD.io VPP.
Unfortunately, VPP crashed with segmentation fault.
Some details:
I used two Dell PowerEdge R430 servers as the Tester and the DUT (Device
Under Test), two 10GbE interfaces of which were interconnected by direct
cables. On the DUT, I used Debian Linux 10.13 with 4.19.0-20-amd64
kernel and the version of FD.io VPP was 22.06. The following packages
were installed: libvppinfra, vpp, vpp-plugin-core, vpp-plugin-dpdk.
I used the following commands to set up Stateful NAT64:
root@p109:~/DUT-settings# cat set-vpp
vppctl set interface state TenGigabitEthernet5/0/0 up
vppctl set interface state TenGigabitEthernet5/0/1 up
vppctl set interface ip address TenGigabitEthernet5/0/0 2001:2::1/64
vppctl set interface ip address TenGigabitEthernet5/0/1 198.19.0.1/24
vppctl ip route add 2001:2::/64 via 2001:2::1 TenGigabitEthernet5/0/0
vppctl ip route add 198.19.0.0/24 via 198.19.0.1 TenGigabitEthernet5/0/1
vppctl set ip neighbor static TenGigabitEthernet5/0/0 2001:2::2
a0:36:9f:74:73:64
vppctl set ip neighbor static TenGigabitEthernet5/0/1 198.19.0.2
a0:36:9f:74:73:66
vppctl set interface nat64 in TenGigabitEthernet5/0/0
vppctl set interface nat64 out TenGigabitEthernet5/0/1
vppctl nat64 add prefix 64:ff9b::/96
vppctl nat64 add pool address 198.19.0.1
As for VPP, first I used two workers, but then I also tried without
workers, using only the main core. Unfortunately, VPP crashed in both
cases, but with somewhat different messages in the syslog. (Previously I
tested both setups with IPv6 packet forwarding and they worked with an
excellent performance.)
The error messages in the syslog when I used two workers:
Nov 7 16:32:02 p109 vnet[2479]: received signal SIGSEGV, PC
0x7fa86f138168, faulting address 0x4f8
Nov 7 16:32:02 p109 vnet[2479]: #0 0x00007fa8b2158137 0x7fa8b2158137
Nov 7 16:32:02 p109 vnet[2479]: #1 0x00007fa8b2086730 0x7fa8b2086730
Nov 7 16:32:02 p109 vnet[2479]: #2 0x00007fa86f138168 0x7fa86f138168
Nov 7 16:32:02 p109 vnet[2479]: #3 0x00007fa86f11d228 0x7fa86f11d228
Nov 7 16:32:02 p109 vnet[2479]: #4 0x00007fa8b20fbe62 0x7fa8b20fbe62
Nov 7 16:32:02 p109 vnet[2479]: #5 0x00007fa8b20fda4f vlib_worker_loop
+ 0x5ff
Nov 7 16:32:02 p109 vnet[2479]: #6 0x00007fa8b2135e79
vlib_worker_thread_fn + 0xa9
Nov 7 16:32:02 p109 vnet[2479]: #7 0x00007fa8b2135290
vlib_worker_thread_bootstrap_fn + 0x50
Nov 7 16:32:02 p109 vnet[2479]: #8 0x00007fa8b207bfa3 start_thread + 0xf3
Nov 7 16:32:02 p109 vnet[2479]: #9 0x00007fa8b1d75eff clone + 0x3f
Nov 7 16:32:02 p109 systemd[1]: vpp.service: Main process exited,
code=killed, status=6/ABRT
The error messages in the syslog when I used only the main core:
Nov 7 16:48:57 p109 vnet[2606]: received signal SIGSEGV, PC
0x7fbe1d24a168, faulting address 0x1a8
Nov 7 16:48:57 p109 vnet[2606]: #0 0x00007fbe6026a137 0x7fbe6026a137
Nov 7 16:48:57 p109 vnet[2606]: #1 0x00007fbe60198730 0x7fbe60198730
Nov 7 16:48:57 p109 vnet[2606]: #2 0x00007fbe1d24a168 0x7fbe1d24a168
Nov 7 16:48:57 p109 vnet[2606]: #3 0x00007fbe1d22f228 0x7fbe1d22f228
Nov 7 16:48:57 p109 vnet[2606]: #4 0x00007fbe6020de62 0x7fbe6020de62
Nov 7 16:48:57 p109 vnet[2606]: #5 0x00007fbe602127d1 vlib_main + 0xd41
Nov 7 16:48:57 p109 vnet[2606]: #6 0x00007fbe6026906a 0x7fbe6026906a
Nov 7 16:48:57 p109 vnet[2606]: #7 0x00007fbe60169964 0x7fbe60169964
Nov 7 16:48:57 p109 systemd[1]: vpp.service: Main process exited,
code=killed, status=6/ABRT
As for the first time I started with quite high load, I suspected that I
exhausted some sort of resources, so I tried with much lower load, but
the same thing happened even when I sent only a single packet.
I used siitperf as Tester: https://github.com/lencsegabor/siitperf
And I followed this methodology:
https://datatracker.ietf.org/doc/html/draft-ietf-bmwg-benchmarking-stateful
Previously my tests were successful with the following stateful NAT64
implementations:
- Jool
- tayga+iptables
- OpenBSD PF
Could you please help me why VPP crashes, and how I could make it work?
Thank you very much for your help in advance!
Best regards,
Gábor Lencse
-=-=-=-=-=-=-=-=-=-=-=-
Links: You receive all messages sent to this group.
View/Reply Online (#22153): https://lists.fd.io/g/vpp-dev/message/22153
Mute This Topic: https://lists.fd.io/mt/94908130/21656
Group Owner: vpp-dev+ow...@lists.fd.io
Unsubscribe: https://lists.fd.io/g/vpp-dev/leave/1480452/21656/631435203/xyzzy
[arch...@mail-archive.com]
-=-=-=-=-=-=-=-=-=-=-=-
