Hi Mechthild,
Unfortunately, my knowledge is not sufficient regarding identification of the
bottleneck in the details of your setup.
Wish you best luck,
Piotr
From: vpp-dev@lists.fd.io <vpp-dev@lists.fd.io> On Behalf Of Mechthild Buescher
via lists.fd.io
Sent: Thursday, December 22, 2022 12:25 PM
To: vpp-dev@lists.fd.io
Subject: Re: [vpp-dev] VPP crashes with
vlib_worker_thread_barrier_syn_int:thread deadlock
Hi Piotr,
Thanks for your hint. We could nail down the problem a bit and would like to
ask for your suggestion on how to solve it.
The problem occurs when there is traffic received on a DPDK-interface and shall
be forwarded to a host interface. Here are the snippets of the configuration
which we think is relevant (note, it's a different setup compared to previous
email, here we have one socket and less cpus):
We configure CPU's 2,17,18 to be isolated:
# cat /proc/cmdline
BOOT_IMAGE=/vmlinuz-5.3.18-150300.59.76-default
root=UUID=439b3b24-9c1d-4b6f-b024-539b50cb7480 rootflags=subvol=@
intel_iommu=on iommu=pt intel_idle.max_cstate=0 processor.max_cstate=0
idle=poll intel_pstate=disable isolcpus=2,3,4,5,6,17,18,19,20,21,22 nohz=on
nohz_full=2,3,4,5,6,17,18,19,20,21,22 rcu_nocbs=2,3,4,5,6,17,18,19,20,21,22
rcu_nocb_poll
irqaffinity=0,1,7,8,9,10,11,12,13,14,15,16,23,24,25,26,27,28,29,30,31
hugepagesz=2M hugepages=2048 hugepagesz=1G hugepages=4 default_hugepagesz=2M
transparent_hugepage=never nosoftlookup nmi_watchdog=0 tsc=reliable
hpet=disable clocksource=tsc skew_tick=1 mce=ignore_ce splash
console=ttyS0,115200 psi=1 audit=1 apparmor=1 security=apparmor
And we use those isolated CPU's for the workers and one non-isolated CPU for
the main-thread:
cpu {
main-core 1
corelist-workers 2,17,18
}
The relevant DPDK-interface is Radio-0:
dpdk {
dev default {
num-rx-queues 3
}
uio-driver vfio-pci
dev 0000:17:00.1 {
name Radio-0
}
:
}
And then we have the following configuration:
set interface state Radio-0 up
create host-interface name Vpp2Host
set interface state host-Vpp2Host up
set interface rx-placement host-Vpp2Host main
create sub-interfaces Radio-0 3092
set interface state Radio-0.3092 up
create sub-interfaces host-Vpp2Host 3092
set interface state host-Vpp2Host.3092 up
set interface l2 bridge Radio-0.3092 3092
set interface l2 bridge host-Vpp2Host.3092 3092
This means, we receive traffic on a DPDK interface and try to forward it via a
L2 bridge to the host. The DPDK interface is on an isolated CPU while the host
interface is on a non-isolated CPU. My suspicion is that this is the problem -
do you agree? Do you have any idea how we can solve this? The Radio-0 interface
is used for OAM via vlan 3092 (this is what you see in the above configuration)
as well as for traffic (untagged), that's why we want to have it on an isolated
CPU.
Thank you for your support,
BR/Mechthild
From: vpp-dev@lists.fd.io<mailto:vpp-dev@lists.fd.io>
<vpp-dev@lists.fd.io<mailto:vpp-dev@lists.fd.io>> on behalf of Bronowski,
PiotrX via lists.fd.io
<piotrx.bronowski=intel....@lists.fd.io<mailto:piotrx.bronowski=intel....@lists.fd.io>>
Date: Wednesday, 21. December 2022 at 17:24
To: vpp-dev@lists.fd.io<mailto:vpp-dev@lists.fd.io>
<vpp-dev@lists.fd.io<mailto:vpp-dev@lists.fd.io>>
Subject: Re: [vpp-dev] VPP crashes with
vlib_worker_thread_barrier_syn_int:thread deadlock
Hi Mechthild,
Your issue is caused by the main thread waiting too long for a worker to
finish. You may examine these lines in src/vlib/threads.h:
171 /* Check for a barrier sync request every 30ms */
172 #define BARRIER_SYNC_DELAY (0.030000)
173
174 #if CLIB_DEBUG > 0
175 /* long barrier timeout, for gdb... */
176 #define BARRIER_SYNC_TIMEOUT (600.1)
177 #else
178 #define BARRIER_SYNC_TIMEOUT (1.0)
179 #endif
Your restart is caused by the timeout defined in these lines. You may increase
it to investigate your issue (of course it is not a fix). After increasing
timeout and being in interactive mode you can issue command "show run" it will
tell you in which node you are spending most of your time and potentially
identify source of your problem. Alternatively, you may use perf tool for that
task.
BR,
Piotr
From: vpp-dev@lists.fd.io<mailto:vpp-dev@lists.fd.io>
<vpp-dev@lists.fd.io<mailto:vpp-dev@lists.fd.io>> On Behalf Of Mechthild
Buescher via lists.fd.io
Sent: Wednesday, December 21, 2022 3:24 PM
To: vpp-dev@lists.fd.io<mailto:vpp-dev@lists.fd.io>
Subject: [vpp-dev] VPP crashes with vlib_worker_thread_barrier_syn_int:thread
deadlock
Hi,
We have a severe problem with VPP - it's cyclic restarting due to the following
error:
Dec 21 09:46:32 843V0N3 vpp[3846]: vlib_worker_thread_barrier_sync_int: worker
thread deadlock
This happens on both servers of the setup and it cannot recover. Can you please
help us to debug this issue?
VPP version:
# vppctl show version
vpp v22.02.0-1~g0d1b46707-dirty built by suse on SUSE at 2022-05-02T09:46:05
which is a built of version 22.02.-1 on SLES 15 SP3 including the dpdk-patch
0001-add-patch-to-disable-source-pruning-in-i40e-driver.patch
The startup.conf:
unix {
nodaemon
log /var/log/vpp/vpp.log
full-coredump
coredump-size unlimited
cli-listen /run/vpp/cli.sock
gid vpp
startup-config /etc/vpp/vpp-static-config.txt
}
api-trace {
on
}
api-segment {
gid vpp
}
socksvr {
socket-name /var/run/vpp/ic-api.sock
}
memory {
main-heap-page-size 1G
}
cpu {
main-core 2
corelist-workers 4,6,42,44,46
}
buffers {
buffers-per-numa 128000
}
dpdk {
dev default {
num-rx-queues 5
}
uio-driver vfio-pci
dev 0000:3b:00.0 {
name Radio-0
}
dev 0000:3b:00.1 {
name Ext-0
}
dev 0000:5e:02.1 {
name NCIC-1-v1
}
}
plugins {
plugin default { disable }
plugin dpdk_plugin.so { enable }
plugin ioam_plugin.so { enable }
plugin perfmon_plugin.so { enable }
plugin tracedump_plugin.so { enable }
plugin l3xc_plugin.so { enable }
plugin ping_plugin.so { enable }
plugin avf_plugin.so { enable }
plugin acl_plugin.so { enable }
plugin svs_plugin.so { enable }
plugin vrrp_plugin.so { enable }
plugin dhcp_plugin.so { enable }
plugin nat_plugin.so { enable }
plugin abf_plugin.so { enable }
plugin lacp_plugin.so { enable }
plugin flowprobe_plugin.so { enable }
}
The log gives:
Dec 21 10:33:27 hostname systemd[1]: Starting Vector Packet Processing
Process...
Dec 21 10:33:27 hostname ic-vpp-service.sh[64758]: +
dpdk_devbind=/usr/local/bin/dpdk-devbind.py
Dec 21 10:33:27 hostname ic-vpp-service.sh[64758]: +
VPP_CONF=/etc/vpp/ic-startup.conf
Dec 21 10:33:27 hostname ic-vpp-service.sh[64760]: ++ grep -v '#'
/etc/vpp/ic-startup.conf
Dec 21 10:33:27 hostname ic-vpp-service.sh[64761]: ++ grep dev
Dec 21 10:33:27 hostname ic-vpp-service.sh[64762]: ++ grep -v default
Dec 21 10:33:27 hostname ic-vpp-service.sh[64763]: ++ sed 's/.*dev //'
Dec 21 10:33:27 hostname ic-vpp-service.sh[64764]: ++ cut '-d ' -f1
Dec 21 10:33:27 hostname ic-vpp-service.sh[64758]: + DEVICES='0000:3b:00.0
Dec 21 10:33:27 hostname ic-vpp-service.sh[64758]: 0000:3b:00.1
Dec 21 10:33:27 hostname ic-vpp-service.sh[64758]: 0000:5e:02.1'
Dec 21 10:33:27 hostname ic-vpp-service.sh[64766]: ++ grep uio-driver
/etc/vpp/ic-startup.conf
Dec 21 10:33:27 hostname ic-vpp-service.sh[64767]: ++ sed 's/.*uio-driver //'
Dec 21 10:33:27 hostname ic-vpp-service.sh[64768]: ++ cut '-d ' -f1
Dec 21 10:33:27 hostname ic-vpp-service.sh[64758]: + DPDK_DRV=vfio-pci
Dec 21 10:33:27 hostname ic-vpp-service.sh[64758]: + '[' --start == --stop ']'
Dec 21 10:33:27 hostname ic-vpp-service.sh[64758]: + for dev in $DEVICES
Dec 21 10:33:27 hostname ic-vpp-service.sh[64770]: ++
/usr/local/bin/dpdk-devbind.py -s
Dec 21 10:33:27 hostname ic-vpp-service.sh[64771]: ++ grep 0000:3b:00.0
Dec 21 10:33:27 hostname ic-vpp-service.sh[64758]: + cdev='0000:3b:00.0
'\''Ethernet Controller XXV710 for 25GbE SFP28 158b'\'' if=Radio-0 drv=i40e
unused=vfio-pci '
Dec 21 10:33:27 hostname ic-vpp-service.sh[64952]: ++ get_drv '0000:3b:00.0
'\''Ethernet Controller XXV710 for 25GbE SFP28 158b'\'' if=Radio-0 drv=i40e
unused=vfio-pci '
Dec 21 10:33:27 hostname ic-vpp-service.sh[64952]: ++ [[ -z 0000:3b:00.0
'Ethernet Controller XXV710 for 25GbE SFP28 158b' if=Radio-0 drv=i40e
unused=vfio-pci ]]
Dec 21 10:33:27 hostname ic-vpp-service.sh[64952]: ++ echo i40e
Dec 21 10:33:27 hostname ic-vpp-service.sh[64758]: + drv=i40e
Dec 21 10:33:27 hostname ic-vpp-service.sh[64758]: + check_intf --start
0000:3b:00.0 ''\''Ethernet' Controller XXV710 for 25GbE SFP28 '158b'\'''
if=Radio-0 drv=i40e unused=vfio-pci
Dec 21 10:33:27 hostname ic-vpp-service.sh[64758]: + action=--start
Dec 21 10:33:27 hostname ic-vpp-service.sh[64758]: + shift
Dec 21 10:33:27 hostname ic-vpp-service.sh[64954]: ++ echo 0000:3b:00.0
''\''Ethernet' Controller XXV710 for 25GbE SFP28 '158b'\''' if=Radio-0 drv=i40e
unused=vfio-pci
Dec 21 10:33:27 hostname ic-vpp-service.sh[64955]: ++ sed 's/.* if=//'
Dec 21 10:33:27 hostname ic-vpp-service.sh[64956]: ++ cut '-d ' -f1
Dec 21 10:33:27 hostname ic-vpp-service.sh[64758]: + name=Radio-0
Dec 21 10:33:27 hostname ic-vpp-service.sh[64758]: + [[ Radio-0 == *\-\v* ]]
Dec 21 10:33:27 hostname ic-vpp-service.sh[64758]: + '[' --start == --stop ']'
Dec 21 10:33:27 hostname ic-vpp-service.sh[64758]: + ip l s Radio-0 down
Dec 21 10:33:27 hostname ic-vpp-service.sh[64758]: +
/usr/local/bin/dpdk-devbind.py -b vfio-pci 0000:3b:00.0
Dec 21 10:33:30 hostname ic-vpp-service.sh[64758]: + for dev in $DEVICES
Dec 21 10:33:30 hostname ic-vpp-service.sh[66019]: ++ grep 0000:3b:00.1
Dec 21 10:33:30 hostname ic-vpp-service.sh[66018]: ++
/usr/local/bin/dpdk-devbind.py -s
Dec 21 10:33:30 hostname ic-vpp-service.sh[64758]: + cdev='0000:3b:00.1
'\''Ethernet Controller XXV710 for 25GbE SFP28 158b'\'' if=Ext-0 drv=i40e
unused=vfio-pci '
Dec 21 10:33:30 hostname ic-vpp-service.sh[66366]: ++ get_drv '0000:3b:00.1
'\''Ethernet Controller XXV710 for 25GbE SFP28 158b'\'' if=Ext-0 drv=i40e
unused=vfio-pci '
Dec 21 10:33:30 hostname ic-vpp-service.sh[66366]: ++ [[ -z 0000:3b:00.1
'Ethernet Controller XXV710 for 25GbE SFP28 158b' if=Ext-0 drv=i40e
unused=vfio-pci ]]
Dec 21 10:33:30 hostname ic-vpp-service.sh[66366]: ++ echo i40e
Dec 21 10:33:30 hostname ic-vpp-service.sh[64758]: + drv=i40e
Dec 21 10:33:30 hostname ic-vpp-service.sh[64758]: + check_intf --start
0000:3b:00.1 ''\''Ethernet' Controller XXV710 for 25GbE SFP28 '158b'\'''
if=Ext-0 drv=i40e unused=vfio-pci
Dec 21 10:33:30 hostname ic-vpp-service.sh[64758]: + action=--start
Dec 21 10:33:30 hostname ic-vpp-service.sh[64758]: + shift
Dec 21 10:33:30 hostname ic-vpp-service.sh[66368]: ++ echo 0000:3b:00.1
''\''Ethernet' Controller XXV710 for 25GbE SFP28 '158b'\''' if=Ext-0 drv=i40e
unused=vfio-pci
Dec 21 10:33:30 hostname ic-vpp-service.sh[66369]: ++ sed 's/.* if=//'
Dec 21 10:33:30 hostname ic-vpp-service.sh[66370]: ++ cut '-d ' -f1
Dec 21 10:33:30 hostname ic-vpp-service.sh[64758]: + name=Ext-0
Dec 21 10:33:30 hostname ic-vpp-service.sh[64758]: + [[ Ext-0 == *\-\v* ]]
Dec 21 10:33:30 hostname ic-vpp-service.sh[64758]: + '[' --start == --stop ']'
Dec 21 10:33:30 hostname ic-vpp-service.sh[64758]: + ip l s Ext-0 down
Dec 21 10:33:30 hostname ic-vpp-service.sh[64758]: +
/usr/local/bin/dpdk-devbind.py -b vfio-pci 0000:3b:00.1
Dec 21 10:33:33 hostname ic-vpp-service.sh[64758]: + for dev in $DEVICES
Dec 21 10:33:33 hostname ic-vpp-service.sh[66986]: ++
/usr/local/bin/dpdk-devbind.py -s
Dec 21 10:33:33 hostname ic-vpp-service.sh[66987]: ++ grep 0000:5e:02.1
Dec 21 10:33:33 hostname ic-vpp-service.sh[64758]: + cdev='0000:5e:02.1
'\''Ethernet Virtual Function 700 Series 154c'\'' if=NCIC-1-v1 drv=iavf
unused=vfio-pci '
Dec 21 10:33:33 hostname ic-vpp-service.sh[67199]: ++ get_drv '0000:5e:02.1
'\''Ethernet Virtual Function 700 Series 154c'\'' if=NCIC-1-v1 drv=iavf
unused=vfio-pci '
Dec 21 10:33:33 hostname ic-vpp-service.sh[67199]: ++ [[ -z '' ]]
Dec 21 10:33:33 hostname ic-vpp-service.sh[67199]: ++ echo iavf
Dec 21 10:33:33 hostname ic-vpp-service.sh[64758]: + drv=iavf
Dec 21 10:33:33 hostname ic-vpp-service.sh[64758]: + check_intf --start
0000:5e:02.1 ''\''Ethernet' Virtual Function 700 Series '154c'\''' if=NCIC-1-v1
drv=iavf unused=vfio-pci
Dec 21 10:33:33 hostname ic-vpp-service.sh[64758]: + action=--start
Dec 21 10:33:33 hostname ic-vpp-service.sh[64758]: + shift
Dec 21 10:33:33 hostname ic-vpp-service.sh[67201]: ++ echo 0000:5e:02.1
''\''Ethernet' Virtual Function 700 Series '154c'\''' if=NCIC-1-v1 drv=iavf
unused=vfio-pci
Dec 21 10:33:33 hostname ic-vpp-service.sh[67202]: ++ sed 's/.* if=//'
Dec 21 10:33:33 hostname ic-vpp-service.sh[67203]: ++ cut '-d ' -f1
Dec 21 10:33:33 hostname ic-vpp-service.sh[64758]: + name=NCIC-1-v1
Dec 21 10:33:33 hostname ic-vpp-service.sh[64758]: + [[ NCIC-1-v1 == *\-\v* ]]
Dec 21 10:33:33 hostname ic-vpp-service.sh[64758]: + parent=NCIC-1
Dec 21 10:33:33 hostname ic-vpp-service.sh[64758]: + vf=1
Dec 21 10:33:33 hostname ic-vpp-service.sh[64758]: + '[' --start == --stop ']'
Dec 21 10:33:33 hostname ic-vpp-service.sh[64758]: + ip l s NCIC-1 vf 1 spoof
off state enable trust on
Dec 21 10:33:33 hostname ic-vpp-service.sh[64758]: +
/usr/local/bin/dpdk-devbind.py -b vfio-pci 0000:5e:02.1
Dec 21 10:33:34 hostname systemd[1]: Started Vector Packet Processing Process.
Dec 21 10:33:37 hostname vnet[67475]: dpdk/cryptodev: dpdk_cryptodev_init: Not
enough cryptodev resources
Dec 21 10:33:37 hostname vnet[67475]: vat-plug/load: vat_plugin_register: adl
plugin not loaded...
Dec 21 10:33:37 hostname vnet[67475]: vat-plug/load: vat_plugin_register:
arping plugin not loaded...
Dec 21 10:33:37 hostname vnet[67475]: vat-plug/load: vat_plugin_register:
builtinurl plugin not loaded...
Dec 21 10:33:37 hostname vnet[67475]: vat-plug/load: vat_plugin_register: cdp
plugin not loaded...
Dec 21 10:33:37 hostname vnet[67475]: vat-plug/load: vat_plugin_register: ct6
plugin not loaded...
Dec 21 10:33:37 hostname vnet[67475]: vat-plug/load: vat_plugin_register: dns
plugin not loaded...
Dec 21 10:33:37 hostname vnet[67475]: vat-plug/load: vat_plugin_register:
geneve plugin not loaded...
Dec 21 10:33:37 hostname vnet[67475]: vat-plug/load: vat_plugin_register: gtpu
plugin not loaded...
Dec 21 10:33:37 hostname vnet[67475]: vat-plug/load: vat_plugin_register:
http_static plugin not loaded...
Dec 21 10:33:37 hostname vnet[67475]: vat-plug/load: vat_plugin_register: ikev2
plugin not loaded...
Dec 21 10:33:37 hostname vnet[67475]: vat-plug/load: vat_plugin_register: l2tp
plugin not loaded...
Dec 21 10:33:37 hostname vnet[67475]: vat-plug/load: vat_plugin_register: lb
plugin not loaded...
Dec 21 10:33:37 hostname vnet[67475]: vat-plug/load: vat_plugin_register_gpe:
lisp_gpe plugin not loaded...
Dec 21 10:33:37 hostname vnet[67475]: vat-plug/load: vat_plugin_register: lldp
plugin not loaded...
Dec 21 10:33:37 hostname vnet[67475]: vat-plug/load: vat_plugin_register:
mactime plugin not loaded...
Dec 21 10:33:37 hostname vnet[67475]: vat-plug/load: vat_plugin_register: mdata
plugin not loaded...
Dec 21 10:33:37 hostname vnet[67475]: vat-plug/load: vat_plugin_register: memif
plugin not loaded...
Dec 21 10:33:37 hostname vnet[67475]: vat-plug/load: vat_plugin_register: nsh
plugin not loaded...
Dec 21 10:33:37 hostname vnet[67475]: vat-plug/load: vat_plugin_register: nsim
plugin not loaded...
Dec 21 10:33:37 hostname vnet[67475]: vat-plug/load: vat_plugin_register:
oddbuf plugin not loaded...
Dec 21 10:33:37 hostname vnet[67475]: vat-plug/load: vat_plugin_register: pppoe
plugin not loaded...
Dec 21 10:33:37 hostname vnet[67475]: vat-plug/load: vat_plugin_register: rdma
plugin not loaded...
Dec 21 10:33:37 hostname vnet[67475]: vat-plug/load: vat_plugin_register: stn
plugin not loaded...
Dec 21 10:33:37 hostname vnet[67475]: vat-plug/load: vat_plugin_register:
tls_openssl plugin not loaded...
Dec 21 10:33:37 hostname vnet[67475]: vat-plug/load: vat_plugin_register:
vmxnet3 plugin not loaded...
Dec 21 10:33:40 hostname vnet[67475]: interface: hw_add_del_mac_address:
vnet_hw_interface_add_del_mac_address: Secondary MAC Addresses not supported
for interface index 0
Dec 21 10:33:40 hostname vnet[67475]: interface: hw_add_del_mac_address:
vnet_hw_interface_add_del_mac_address: Secondary MAC Addresses not supported
for interface index 0
Dec 21 10:33:40 hostname vnet[67475]: interface: hw_add_del_mac_address:
vnet_hw_interface_add_del_mac_address: Secondary MAC Addresses not supported
for interface index 0
Dec 21 10:33:40 hostname vnet[67475]: interface: hw_add_del_mac_address:
vnet_hw_interface_add_del_mac_address: Secondary MAC Addresses not supported
for interface index 0
Dec 21 10:33:40 hostname vnet[67475]: interface: hw_add_del_mac_address:
vnet_hw_interface_add_del_mac_address: Secondary MAC Addresses not supported
for interface index 0
Dec 21 10:33:40 hostname vnet[67475]: interface: hw_add_del_mac_address:
vnet_hw_interface_add_del_mac_address: Secondary MAC Addresses not supported
for interface index 0
Dec 21 10:33:40 hostname vnet[67475]: interface: hw_add_del_mac_address:
vnet_hw_interface_add_del_mac_address: Secondary MAC Addresses not supported
for interface index 0
Dec 21 10:33:40 hostname vnet[67475]: interface: hw_add_del_mac_address:
vnet_hw_interface_add_del_mac_address: Secondary MAC Addresses not supported
for interface index 0
Dec 21 10:33:40 hostname vnet[67475]: interface: hw_add_del_mac_address:
vnet_hw_interface_add_del_mac_address: Secondary MAC Addresses not supported
for interface index 0
Dec 21 10:33:40 hostname vnet[67475]: interface: hw_add_del_mac_address:
vnet_hw_interface_add_del_mac_address: Secondary MAC Addresses not supported
for interface index 0
Dec 21 10:33:40 hostname vnet[67475]: interface: hw_add_del_mac_address:
vnet_hw_interface_add_del_mac_address: Secondary MAC Addresses not supported
for interface index 0
Dec 21 10:33:40 hostname vnet[67475]: interface: hw_add_del_mac_address:
vnet_hw_interface_add_del_mac_address: Secondary MAC Addresses not supported
for interface index 0
Dec 21 10:33:40 hostname vnet[67475]: interface: hw_add_del_mac_address:
vnet_hw_interface_add_del_mac_address: Secondary MAC Addresses not supported
for interface index 0
Dec 21 10:33:40 hostname vnet[67475]: interface: hw_add_del_mac_address:
vnet_hw_interface_add_del_mac_address: Secondary MAC Addresses not supported
for interface index 0
Dec 21 10:33:40 hostname vnet[67475]: interface: hw_add_del_mac_address:
vnet_hw_interface_add_del_mac_address: Secondary MAC Addresses not supported
for interface index 0
Dec 21 10:33:40 hostname vnet[67475]: interface: hw_add_del_mac_address:
vnet_hw_interface_add_del_mac_address: Secondary MAC Addresses not supported
for interface index 0
Dec 21 10:33:40 hostname vnet[67475]: interface: hw_add_del_mac_address:
vnet_hw_interface_add_del_mac_address: Secondary MAC Addresses not supported
for interface index 0
Dec 21 10:33:40 hostname vnet[67475]: interface: hw_add_del_mac_address:
vnet_hw_interface_add_del_mac_address: Secondary MAC Addresses not supported
for interface index 0
Dec 21 10:33:40 hostname vnet[67475]: interface: hw_add_del_mac_address:
vnet_hw_interface_add_del_mac_address: Secondary MAC Addresses not supported
for interface index 0
Dec 21 10:33:40 hostname vnet[67475]: interface: hw_add_del_mac_address:
vnet_hw_interface_add_del_mac_address: Secondary MAC Addresses not supported
for interface index 0
Dec 21 10:33:40 hostname vnet[67475]: interface: hw_add_del_mac_address:
vnet_hw_interface_add_del_mac_address: Secondary MAC Addresses not supported
for interface index 0
Dec 21 10:33:40 hostname vnet[67475]: interface: hw_add_del_mac_address:
vnet_hw_interface_add_del_mac_address: Secondary MAC Addresses not supported
for interface index 0
Dec 21 10:33:40 hostname vnet[67475]: interface: hw_add_del_mac_address:
vnet_hw_interface_add_del_mac_address: Secondary MAC Addresses not supported
for interface index 0
Dec 21 10:33:40 hostname vnet[67475]: interface: hw_add_del_mac_address:
vnet_hw_interface_add_del_mac_address: Secondary MAC Addresses not supported
for interface index 0
Dec 21 10:33:40 hostname vnet[67475]: interface: hw_add_del_mac_address:
vnet_hw_interface_add_del_mac_address: Secondary MAC Addresses not supported
for interface index 0
Dec 21 10:33:40 hostname vnet[67475]: interface: hw_add_del_mac_address:
vnet_hw_interface_add_del_mac_address: Secondary MAC Addresses not supported
for interface index 0
Dec 21 10:33:40 hostname vnet[67475]: interface: hw_add_del_mac_address:
vnet_hw_interface_add_del_mac_address: Secondary MAC Addresses not supported
for interface index 0
Dec 21 10:33:40 hostname vnet[67475]: interface: hw_add_del_mac_address:
vnet_hw_interface_add_del_mac_address: Secondary MAC Addresses not supported
for interface index 0
Dec 21 10:33:40 hostname vnet[67475]: interface: hw_add_del_mac_address:
vnet_hw_interface_add_del_mac_address: Secondary MAC Addresses not supported
for interface index 0
Dec 21 10:33:40 hostname vnet[67475]: interface: hw_add_del_mac_address:
vnet_hw_interface_add_del_mac_address: Secondary MAC Addresses not supported
for interface index 0
Dec 21 10:33:40 hostname vnet[67475]: vrrp_vr_transition:282: VR [0]
sw_if_index 2 VR ID 11 IPv4 transitioning to Backup
Dec 21 10:33:40 hostname vnet[67475]: vrrp_vr_transition_vmac:120: Deleting
virtual MAC address 00:00:5e:00:01:0b on hardware interface 2
Dec 21 10:33:40 hostname vnet[67475]: vrrp_vr_start_stop:768: 1 VRs configured,
1 VRs running
Dec 21 10:33:40 hostname vnet[67475]: interface: hw_add_del_mac_address:
vnet_hw_interface_add_del_mac_address: Secondary MAC Addresses not supported
for interface index 0
Dec 21 10:33:40 hostname vnet[67475]: interface: hw_add_del_mac_address:
vnet_hw_interface_add_del_mac_address: Secondary MAC Addresses not supported
for interface index 0
Dec 21 10:33:40 hostname vnet[67475]: interface: hw_add_del_mac_address:
vnet_hw_interface_add_del_mac_address: Secondary MAC Addresses not supported
for interface index 0
Dec 21 10:33:40 hostname vnet[67475]: vrrp_vr_transition:282: VR [1]
sw_if_index 24 VR ID 207 IPv4 transitioning to Backup
Dec 21 10:33:40 hostname vnet[67475]: vrrp_vr_transition_vmac:120: Deleting
virtual MAC address 00:00:5e:00:01:cf on hardware interface 2
Dec 21 10:33:40 hostname vnet[67475]: vrrp_vr_start_stop:768: 2 VRs configured,
2 VRs running
Dec 21 10:33:40 hostname vnet[67475]: interface: hw_add_del_mac_address:
vnet_hw_interface_add_del_mac_address: Secondary MAC Addresses not supported
for interface index 0
Dec 21 10:33:40 hostname vnet[67475]: interface: hw_add_del_mac_address:
vnet_hw_interface_add_del_mac_address: Secondary MAC Addresses not supported
for interface index 0
Dec 21 10:33:40 hostname vnet[67475]: interface: hw_add_del_mac_address:
vnet_hw_interface_add_del_mac_address: Secondary MAC Addresses not supported
for interface index 0
Dec 21 10:33:40 hostname vnet[67475]: vrrp_vr_transition:282: VR [2]
sw_if_index 25 VR ID 202 IPv4 transitioning to Backup
Dec 21 10:33:40 hostname vnet[67475]: vrrp_vr_transition_vmac:120: Deleting
virtual MAC address 00:00:5e:00:01:ca on hardware interface 2
Dec 21 10:33:40 hostname vnet[67475]: vrrp_vr_start_stop:768: 3 VRs configured,
3 VRs running
Dec 21 10:33:40 hostname vnet[67475]: interface: hw_add_del_mac_address:
vnet_hw_interface_add_del_mac_address: Secondary MAC Addresses not supported
for interface index 0
Dec 21 10:33:40 hostname vnet[67475]: interface: hw_add_del_mac_address:
vnet_hw_interface_add_del_mac_address: Secondary MAC Addresses not supported
for interface index 0
Dec 21 10:33:40 hostname vnet[67475]: interface: hw_add_del_mac_address:
vnet_hw_interface_add_del_mac_address: Secondary MAC Addresses not supported
for interface index 0
Dec 21 10:33:40 hostname vnet[67475]: vrrp_vr_transition:282: VR [3]
sw_if_index 26 VR ID 201 IPv4 transitioning to Backup
Dec 21 10:33:40 hostname vnet[67475]: vrrp_vr_transition_vmac:120: Deleting
virtual MAC address 00:00:5e:00:01:c9 on hardware interface 2
Dec 21 10:33:40 hostname vnet[67475]: vrrp_vr_start_stop:768: 4 VRs configured,
4 VRs running
Dec 21 10:33:44 hostname vnet[67475]: vrrp_vr_transition:282: VR [0]
sw_if_index 2 VR ID 11 IPv4 transitioning to Master
Dec 21 10:33:44 hostname vnet[67475]: vrrp_vr_transition_addrs:237: Adding VR
addresses on sw_if_index 2
Dec 21 10:33:44 hostname vnet[67475]: vrrp_vr_transition_vmac:120: Adding
virtual MAC address 00:00:5e:00:01:0b on hardware interface 2
Dec 21 10:33:44 hostname vnet[67475]: vrrp_vr_transition:282: VR [1]
sw_if_index 24 VR ID 207 IPv4 transitioning to Master
Dec 21 10:33:44 hostname vnet[67475]: vrrp_vr_transition_addrs:237: Adding VR
addresses on sw_if_index 24
Dec 21 10:33:44 hostname vnet[67475]: vrrp_vr_transition_vmac:120: Adding
virtual MAC address 00:00:5e:00:01:cf on hardware interface 2
Dec 21 10:33:44 hostname vnet[67475]: vrrp_vr_transition:282: VR [2]
sw_if_index 25 VR ID 202 IPv4 transitioning to Master
Dec 21 10:33:44 hostname vnet[67475]: vrrp_vr_transition_addrs:237: Adding VR
addresses on sw_if_index 25
Dec 21 10:33:44 hostname vnet[67475]: vrrp_vr_transition_vmac:120: Adding
virtual MAC address 00:00:5e:00:01:ca on hardware interface 2
Dec 21 10:33:44 hostname vnet[67475]: vrrp_vr_transition:282: VR [3]
sw_if_index 26 VR ID 201 IPv4 transitioning to Master
Dec 21 10:33:44 hostname vnet[67475]: vrrp_vr_transition_addrs:237: Adding VR
addresses on sw_if_index 26
Dec 21 10:33:44 hostname vnet[67475]: vrrp_vr_transition_vmac:120: Adding
virtual MAC address 00:00:5e:00:01:c9 on hardware interface 2
Dec 21 10:34:07 hostname vnet[67475]: vrrp_vr_start_stop:732: Attempting to
start already started VR ([0] sw_if_index 2 VR ID 11 IPv4)
Dec 21 10:34:07 hostname vnet[67475]: vrrp_vr_start_stop:732: Attempting to
start already started VR ([1] sw_if_index 24 VR ID 207 IPv4)
Dec 21 10:34:07 hostname vnet[67475]: vrrp_vr_start_stop:732: Attempting to
start already started VR ([2] sw_if_index 25 VR ID 202 IPv4)
Dec 21 10:34:07 hostname vnet[67475]: vrrp_vr_start_stop:732: Attempting to
start already started VR ([3] sw_if_index 26 VR ID 201 IPv4)
Dec 21 10:34:36 hostname vpp[67475]: vlib_worker_thread_barrier_sync_int:
worker thread deadlock
Dec 21 10:34:36 hostname vnet[67475]: received signal SIGABRT, PC 0x7f7207c5acdb
Dec 21 10:34:36 hostname vnet[67475]: #0 0x00007f7208ea3234
unix_signal_handler + 0x124
Dec 21 10:34:36 hostname vnet[67475]: #1 0x00007f72087e68c0 0x7f72087e68c0
Dec 21 10:34:36 hostname vnet[67475]: #2 0x00007f7207c5acdb gsignal + 0x10d
Dec 21 10:34:36 hostname vnet[67475]: #3 0x00007f7207c5c375 abort + 0x177
Dec 21 10:34:36 hostname vnet[67475]: #4 0x0000000000407643 0x407643
Dec 21 10:34:36 hostname vnet[67475]: #5 0x00007f7208e89541
vlib_worker_thread_barrier_sync_int + 0x451
Dec 21 10:34:36 hostname vnet[67475]: #6 0x00007f720a5aa3af
vl_api_rpc_call_main_thread + 0x3cf
Dec 21 10:34:36 hostname vnet[67475]: #7 0x00007f7209a95c1b arp_reply + 0x7eb
Dec 21 10:34:36 hostname vnet[67475]: #8 0x00007f7208e50b65
dispatch_pending_node + 0x135
Dec 21 10:34:36 hostname vnet[67475]: #9 0x00007f7208e55214 vlib_main + 0xd34
Dec 21 10:34:36 hostname vnet[67475]: #10 0x00007f7208ea2288 thread0 + 0x28
Dec 21 10:34:36 hostname vnet[67475]: #11 0x00007f72083b600c 0x7f72083b600c
Dec 21 10:34:49 hostname systemd[1]: vpp.service: Main process exited,
code=dumped, status=6/ABRT
We have our own controller which uses the API to do some dynamic configuration
of VPP - apart from different routes in different tables, it configures VRRP as
follows:
# vppctl show vrrp vr
[0] sw_if_index 2 VR ID 11 IPv4
state Master flags: preempt yes accept yes unicast no
priority: configured 200 adjusted 200
timers: adv interval 100 master adv 100 skew 21 master down 321
virtual MAC 00:00:5e:00:01:0b
addresses 10.95.255.2
peer addresses
tracked interfaces
[1] sw_if_index 24 VR ID 207 IPv4
state Master flags: preempt yes accept yes unicast no
priority: configured 200 adjusted 200
timers: adv interval 100 master adv 100 skew 21 master down 321
virtual MAC 00:00:5e:00:01:cf
addresses 10.94.131.34
peer addresses
tracked interfaces
[2] sw_if_index 25 VR ID 202 IPv4
state Master flags: preempt yes accept yes unicast no
priority: configured 200 adjusted 200
timers: adv interval 100 master adv 100 skew 21 master down 321
virtual MAC 00:00:5e:00:01:ca
addresses 10.94.29.90
peer addresses
tracked interfaces
[3] sw_if_index 26 VR ID 201 IPv4
state Master flags: preempt yes accept yes unicast no
priority: configured 200 adjusted 200
timers: adv interval 100 master adv 100 skew 21 master down 321
virtual MAC 00:00:5e:00:01:c9
addresses 10.95.250.3
peer addresses
tracked interfaces
The IP addresses of the VRRP master are:
# vppctl show int addr
Ext-0 (up):
L3 10.95.255.3/29 ip4 table-id 1 fib-idx 4
L3 10.95.255.2/29 ip4 table-id 1 fib-idx 4
Ext-0.507 (up):
L3 10.94.131.35/29 ip4 table-id 2 fib-idx 6
L3 10.94.131.34/29 ip4 table-id 2 fib-idx 6
Ext-0.506 (up):
L3 10.94.29.91/29 ip4 table-id 3 fib-idx 7
L3 10.94.29.90/29 ip4 table-id 3 fib-idx 7
Ext-0.501 (up):
L3 10.95.250.4/29 ip4 table-id 4 fib-idx 8
L3 10.95.250.3/29 ip4 table-id 4 fib-idx 8
I don't have gdb on the affected system. On the build server, I only see:
Core was generated by `/usr/bin/vpp -c /etc/vpp/ic-startup.conf'.
Program terminated with signal SIGABRT, Aborted.
#0 0x00007f18c5b23cdb in ?? ()
[Current thread is 1 (LWP 14555)]
(gdb) list
91 vat_plugin_hash_create ();
92 #endif
93
94 if (!vlib_plugin_path)
95 vpp_find_plugin_path ();
96 }
97
98 /*
99 * Default path for runtime data
100 */
(gdb) bt
#0 0x00007f18c5b23cdb in ?? ()
Backtrace stopped: Cannot access memory at address 0x7f18c116cac0
Any ideas on what to check or change to get rid of the cyclic vpp restart?
Any help is appreciated,
Thank you,
BR/Mechthild
--------------------------------------------------------------
Intel Research and Development Ireland Limited
Registered in Ireland
Registered Office: Collinstown Industrial Park, Leixlip, County Kildare
Registered Number: 308263
This e-mail and any attachments may contain confidential material for the sole
use of the intended recipient(s). Any review or distribution by others is
strictly prohibited. If you are not the intended recipient, please contact the
sender and delete all copies.
--------------------------------------------------------------
Intel Research and Development Ireland Limited
Registered in Ireland
Registered Office: Collinstown Industrial Park, Leixlip, County Kildare
Registered Number: 308263
This e-mail and any attachments may contain confidential material for the sole
use of the intended recipient(s). Any review or distribution by others is
strictly prohibited. If you are not the intended recipient, please contact the
sender and delete all copies.
-=-=-=-=-=-=-=-=-=-=-=-
Links: You receive all messages sent to this group.
View/Reply Online (#22390): https://lists.fd.io/g/vpp-dev/message/22390
Mute This Topic: https://lists.fd.io/mt/95806883/21656
Group Owner: vpp-dev+ow...@lists.fd.io
Unsubscribe: https://lists.fd.io/g/vpp-dev/leave/1480452/21656/631435203/xyzzy
[arch...@mail-archive.com]
-=-=-=-=-=-=-=-=-=-=-=-
