Hi All, We have VPP 21.01 installed on a Centos8 server in our setup. The server is configured with 2 different MUX VLAN. This server is connected to another server through gateway machine for communication. This VPP server was not able to talk to the other server. From investigation, it was found out that default route that is used to talk to the other server was not there. We added default route, but communication still did not work. When we pinged the default route address, we saw high latency and packet drops. While looking at the packet drop , we started seeing VPP crash on the machine. Following are the vpp crash logs:
Jun 7 11:33:27 J3SGISNCIGM03 vnet[42797]: ip4_neighbor_advertise:81: Sending GARP for IP4 address 0.0.0.0 on sw_if_idex 6 Jun 7 11:33:27 J3SGISNCIGM03 vnet[42797]: ip4_neighbor_advertise:81: Sending GARP for IP4 address 0.0.0.0 on sw_if_idex 8 Jun 7 11:33:27 J3SGISNCIGM03 vnet[42797]: ip6_neighbor_advertise:52: Sending unsolicitated NA IP6 address 2001:5b0:ffff:701:b883:31f:19e:7888 on sw_if_idex 8 Jun 7 11:33:27 J3SGISNCIGM03 vnet[42797]: ip4_neighbor_advertise:81: Sending GARP for IP4 address 0.0.51.51 on sw_if_idex 9 Jun 7 11:33:27 J3SGISNCIGM03 vnet[42797]: ip6_neighbor_advertise:52: Sending unsolicitated NA IP6 address 2001:5b0:ffff:502:b883:31f:39e:7888 on sw_if_idex 9 Jun 7 11:33:27 J3SGISNCIGM03 vnet[42797]: ip4_neighbor_advertise:81: Sending GARP for IP4 address 0.0.51.51 on sw_if_idex 7 Jun 7 11:33:27 J3SGISNCIGM03 vnet[42797]: ip6_neighbor_advertise:52: Sending unsolicitated NA IP6 address 2001:5b0:ffff:501:b883:31f:19e:7888 on sw_if_idex 7 Jun 7 11:33:28 J3SGISNCIGM03 phc2sys[3144]: [521231.559] [0:nic0] CLOCK_REALTIME phc offset 151 s0 freq +0 delay 1320 Jun 7 11:33:28 J3SGISNCIGM03 vnet[42797]: received signal SIGABRT, PC 0x7ff405d5e70f Jun 7 11:33:28 J3SGISNCIGM03 vnet[42797]: #0 0x00007ff407759358 0x7ff407759358 Jun 7 11:33:28 J3SGISNCIGM03 vnet[42797]: #1 0x00007ff407055dd0 0x7ff407055dd0 Jun 7 11:33:28 J3SGISNCIGM03 vnet[42797]: #2 0x00007ff405d5e70f gsignal + 0x10f Jun 7 11:33:28 J3SGISNCIGM03 vnet[42797]: #3 0x00007ff405d48b25 abort + 0x127 Jun 7 11:33:28 J3SGISNCIGM03 vnet[42797]: #4 0x00007ff406bea636 0x7ff406bea636 Jun 7 11:33:28 J3SGISNCIGM03 vnet[42797]: #5 0x00007ff406bf2435 vec_resize_allocate_memory + 0x285 Jun 7 11:33:28 J3SGISNCIGM03 vnet[42797]: #6 0x00007ff40771144d vlib_put_next_frame + 0xbd Jun 7 11:33:28 J3SGISNCIGM03 vnet[42797]: #7 0x00007ff408a021ec 0x7ff408a021ec Jun 7 11:33:28 J3SGISNCIGM03 vnet[42797]: #8 0x00007ff40771168a 0x7ff40771168a Jun 7 11:33:28 J3SGISNCIGM03 vnet[42797]: #9 0x00007ff40771631e vlib_main + 0xe4e Jun 7 11:33:28 J3SGISNCIGM03 vnet[42797]: #10 0x00007ff407758796 0x7ff407758796 Jun 7 11:33:28 J3SGISNCIGM03 vnet[42797]: #11 0x00007ff406c2c928 0x7ff406c2c928 Jun 7 11:33:28 J3SGISNCIGM03 ptp4l[3143]: [521231.807] [0:nic0] rms 9 max 24 freq +5834 +/- 28 delay 581 +/- 4 Jun 7 11:33:29 J3SGISNCIGM03 systemd[1]: vpp.service: Main process exited, code=killed, status=6/ABRT Jun 7 11:33:29 J3SGISNCIGM03 systemd[1]: vpp.service: Failed with result 'signal'. Jun 7 11:30:24 J3SGISNCIGM03 systemd[1]: vpp.service: Main process exited, code=killed, status=6/ABRT Jun 7 11:30:24 J3SGISNCIGM03 systemd[1]: vpp.service: Failed with result 'signal'. Jun 7 11:30:29 J3SGISNCIGM03 systemd[1]: vpp.service: Service RestartSec=5s expired, scheduling restart. Jun 7 11:30:29 J3SGISNCIGM03 systemd[1]: vpp.service: Scheduled restart job, restart counter is at 44. Jun 7 11:30:29 J3SGISNCIGM03 vpp[42367]: /usr/bin/vpp[42367]: vlib_sort_init_exit_functions:161: order constraint fcn 'dns_init' not found Jun 7 11:30:29 J3SGISNCIGM03 /usr/bin/vpp[42367]: vlib_sort_init_exit_functions:161: order constraint fcn 'dns_init' not found Jun 7 11:30:29 J3SGISNCIGM03 vpp[42367]: /usr/bin/vpp[42367]: vnet_feature_arc_init:272: feature node 'nat44-in2out-output' not found (before 'ip4-dvr-reinject', arc 'ip4-output') Jun 7 11:30:29 J3SGISNCIGM03 /usr/bin/vpp[42367]: vnet_feature_arc_init:272: feature node 'nat44-in2out-output' not found (before 'ip4-dvr-reinject', arc 'ip4-output') Jun 7 11:30:29 J3SGISNCIGM03 vpp[42367]: /usr/bin/vpp[42367]: dpdk: Unsupported PCI device 0x14e4:0x1657 found at PCI address 0000:02:00.0 Jun 7 11:30:29 J3SGISNCIGM03 /usr/bin/vpp[42367]: dpdk: Unsupported PCI device 0x14e4:0x1657 found at PCI address 0000:02:00.0 Jun 7 11:30:29 J3SGISNCIGM03 vpp[42367]: /usr/bin/vpp[42367]: dpdk: Unsupported PCI device 0x14e4:0x1657 found at PCI address 0000:02:00.1 Jun 7 11:30:29 J3SGISNCIGM03 /usr/bin/vpp[42367]: dpdk: Unsupported PCI device 0x14e4:0x1657 found at PCI address 0000:02:00.1 Jun 7 11:30:29 J3SGISNCIGM03 vpp[42367]: /usr/bin/vpp[42367]: dpdk: Unsupported PCI device 0x14e4:0x1657 found at PCI address 0000:02:00.2 Jun 7 11:30:29 J3SGISNCIGM03 /usr/bin/vpp[42367]: dpdk: Unsupported PCI device 0x14e4:0x1657 found at PCI address 0000:02:00.2 Jun 7 11:30:29 J3SGISNCIGM03 vpp[42367]: /usr/bin/vpp[42367]: dpdk: Unsupported PCI device 0x14e4:0x1657 found at PCI address 0000:02:00.3 Jun 7 11:30:29 J3SGISNCIGM03 /usr/bin/vpp[42367]: dpdk: Unsupported PCI device 0x14e4:0x1657 found at PCI address 0000:02:00.3 Jun 7 11:30:35 J3SGISNCIGM03 systemd[1]: vpp.service: Main process exited, code=killed, status=6/ABRT Jun 7 11:30:35 J3SGISNCIGM03 systemd[1]: vpp.service: Failed with result 'signal'. Jun 7 11:30:40 J3SGISNCIGM03 systemd[1]: vpp.service: Service RestartSec=5s expired, scheduling restart. Jun 7 11:30:40 J3SGISNCIGM03 systemd[1]: vpp.service: Scheduled restart job, restart counter is at 45. Jun 7 11:30:40 J3SGISNCIGM03 vpp[42447]: /usr/bin/vpp[42447]: vlib_sort_init_exit_functions:161: order constraint fcn 'dns_init' not found Jun 7 11:30:40 J3SGISNCIGM03 /usr/bin/vpp[42447]: vlib_sort_init_exit_functions:161: order constraint fcn 'dns_init' not found Jun 7 11:30:40 J3SGISNCIGM03 vpp[42447]: /usr/bin/vpp[42447]: vnet_feature_arc_init:272: feature node 'nat44-in2out-output' not found (before 'ip4-dvr-reinject', arc 'ip4-output') Jun 7 11:30:40 J3SGISNCIGM03 /usr/bin/vpp[42447]: vnet_feature_arc_init:272: feature node 'nat44-in2out-output' not found (before 'ip4-dvr-reinject', arc 'ip4-output') Jun 7 11:30:40 J3SGISNCIGM03 vpp[42447]: /usr/bin/vpp[42447]: dpdk: Unsupported PCI device 0x14e4:0x1657 found at PCI address 0000:02:00.0 Jun 7 11:30:40 J3SGISNCIGM03 /usr/bin/vpp[42447]: dpdk: Unsupported PCI device 0x14e4:0x1657 found at PCI address 0000:02:00.0 Jun 7 11:30:40 J3SGISNCIGM03 vpp[42447]: /usr/bin/vpp[42447]: dpdk: Unsupported PCI device 0x14e4:0x1657 found at PCI address 0000:02:00.1 Jun 7 11:30:40 J3SGISNCIGM03 /usr/bin/vpp[42447]: dpdk: Unsupported PCI device 0x14e4:0x1657 found at PCI address 0000:02:00.1 Jun 7 11:30:40 J3SGISNCIGM03 vpp[42447]: /usr/bin/vpp[42447]: dpdk: Unsupported PCI device 0x14e4:0x1657 found at PCI address 0000:02:00.2 Jun 7 11:30:40 J3SGISNCIGM03 /usr/bin/vpp[42447]: dpdk: Unsupported PCI device 0x14e4:0x1657 found at PCI address 0000:02:00.2 Jun 7 11:30:40 J3SGISNCIGM03 vpp[42447]: /usr/bin/vpp[42447]: dpdk: Unsupported PCI device 0x14e4:0x1657 found at PCI address 0000:02:00.3 Jun 7 11:30:40 J3SGISNCIGM03 /usr/bin/vpp[42447]: dpdk: Unsupported PCI device 0x14e4:0x1657 found at PCI address 0000:02:00.3 Jun 7 11:30:46 J3SGISNCIGM03 systemd[1]: vpp.service: Main process exited, code=killed, status=6/ABRT Jun 7 11:30:46 J3SGISNCIGM03 systemd[1]: vpp.service: Failed with result 'signal'. Jun 7 11:30:51 J3SGISNCIGM03 systemd[1]: vpp.service: Service RestartSec=5s expired, scheduling restart. Jun 7 11:30:51 J3SGISNCIGM03 systemd[1]: vpp.service: Scheduled restart job, restart counter is at 46. Jun 7 11:30:51 J3SGISNCIGM03 vpp[42506]: /usr/bin/vpp[42506]: vlib_sort_init_exit_functions:161: order constraint fcn 'dns_init' not found Jun 7 11:30:51 J3SGISNCIGM03 /usr/bin/vpp[42506]: vlib_sort_init_exit_functions:161: order constraint fcn 'dns_init' not found Jun 7 11:30:51 J3SGISNCIGM03 vpp[42506]: /usr/bin/vpp[42506]: vnet_feature_arc_init:272: feature node 'nat44-in2out-output' not found (before 'ip4-dvr-reinject', arc 'ip4-output') Jun 7 11:30:51 J3SGISNCIGM03 /usr/bin/vpp[42506]: vnet_feature_arc_init:272: feature node 'nat44-in2out-output' not found (before 'ip4-dvr-reinject', arc 'ip4-output') Jun 7 11:30:51 J3SGISNCIGM03 vpp[42506]: /usr/bin/vpp[42506]: dpdk: Unsupported PCI device 0x14e4:0x1657 found at PCI address 0000:02:00.0 Jun 7 11:30:51 J3SGISNCIGM03 /usr/bin/vpp[42506]: dpdk: Unsupported PCI device 0x14e4:0x1657 found at PCI address 0000:02:00.0 Jun 7 11:30:51 J3SGISNCIGM03 vpp[42506]: /usr/bin/vpp[42506]: dpdk: Unsupported PCI device 0x14e4:0x1657 found at PCI address 0000:02:00.1 Jun 7 11:30:51 J3SGISNCIGM03 /usr/bin/vpp[42506]: dpdk: Unsupported PCI device 0x14e4:0x1657 found at PCI address 0000:02:00.1 Jun 7 11:30:51 J3SGISNCIGM03 vpp[42506]: /usr/bin/vpp[42506]: dpdk: Unsupported PCI device 0x14e4:0x1657 found at PCI address 0000:02:00.2 Jun 7 11:30:51 J3SGISNCIGM03 /usr/bin/vpp[42506]: dpdk: Unsupported PCI device 0x14e4:0x1657 found at PCI address 0000:02:00.2 Jun 7 11:30:51 J3SGISNCIGM03 vpp[42506]: /usr/bin/vpp[42506]: dpdk: Unsupported PCI device 0x14e4:0x1657 found at PCI address 0000:02:00.3 Jun 7 11:30:51 J3SGISNCIGM03 /usr/bin/vpp[42506]: dpdk: Unsupported PCI device 0x14e4:0x1657 found at PCI address 0000:02:00.3 Jun 7 11:30:57 J3SGISNCIGM03 systemd[1]: vpp.service: Main process exited, code=killed, status=6/ABRT Jun 7 11:30:57 J3SGISNCIGM03 systemd[1]: vpp.service: Failed with result 'signal'. We had to reboot the server to recover. Reboot fixed the ping drop and high latency issue and vpp crashing. I have attached the startup.conf , vcl.conf and mux_ipddr.cfg along with this post . Can someone please let us know why did the crash happen and how can we prevent VPP from crashing . Thanks and Regards. Ashish
mux_ipaddr.cfg
Description: Binary data
startup _1_.conf
Description: Binary data
vcl.conf
Description: Binary data
-=-=-=-=-=-=-=-=-=-=-=- Links: You receive all messages sent to this group. View/Reply Online (#19568): https://lists.fd.io/g/vpp-dev/message/19568 Mute This Topic: https://lists.fd.io/mt/83553727/21656 Group Owner: vpp-dev+ow...@lists.fd.io Unsubscribe: https://lists.fd.io/g/vpp-dev/unsub [arch...@mail-archive.com] -=-=-=-=-=-=-=-=-=-=-=-
