After upgrading from Open vSwitch 3.1.2-4 to 3.2.0-2 we've been
consistently seeing new OVS crashes when setting QoS on ports. Both
packages were taken from the Debian distribution (
https://packages.debian.org/source/sid/openvswitch) we're running on. From
the core dump we're seeing the following backtrace:
# gdb --batch -ex bt /usr/sbin/ovs-vswitchd /core
[New LWP 67669]
[New LWP 67682]
[New LWP 67681]
[New LWP 67671]
[New LWP 67679]
[New LWP 67680]
[Thread debugging using libthread_db enabled]
Using host libthread_db library "/lib/x86_64-linux-gnu/libthread_db.so.1".
Core was generated by `ovs-vswitchd unix:/tmp/ovs/db.sock -vconsole:emer
-vsyslog:err -vfile:info --ml'.
Program terminated with signal SIGABRT, Aborted.
#0 0x00007fcacecbb0fc in ?? () from /lib/x86_64-linux-gnu/libc.so.6
[Current thread is 1 (Thread 0x7fcacf4cfa80 (LWP 67669))]
#0 0x00007fcacecbb0fc in ?? () from /lib/x86_64-linux-gnu/libc.so.6
#1 0x00007fcacec6d472 in raise () from /lib/x86_64-linux-gnu/libc.so.6
#2 0x00007fcacec574b2 in abort () from /lib/x86_64-linux-gnu/libc.so.6
#3 0x0000560787952c7e in ovs_abort_valist (err_no=<optimized out>,
format=<optimized out>, args=args@entry=0x7ffd14c3bce0) at ../lib/util.c:447
#4 0x0000560787952d14 in ovs_abort (err_no=<optimized out>,
format=format@entry=0x5607879f3ec7 "%s: pthread_%s_%s failed") at
../lib/util.c:439
#5 0x000056078791ee11 in ovs_mutex_lock_at (l_=l_@entry=0x56078934c6c8,
where=where@entry=0x5607879f864b "../lib/netdev-linux.c:2575") at
../lib/ovs-thread.c:76
#6 0x000056078796d03d in netdev_linux_get_speed (netdev_=0x56078934c640,
current=0x7ffd14c3be64, max=0x7ffd14c3be1c) at ../lib/netdev-linux.c:2575
#7 0x00005607878c04f3 in netdev_get_speed (netdev=netdev@entry=0x56078934c640,
current=current@entry=0x7ffd14c3be64, max=0x7ffd14c3be1c, max@entry=0x0) at
../lib/netdev.c:1175
#8 0x0000560787968d67 in htb_parse_qdisc_details__
(netdev=netdev@entry=0x56078934c640,
details=details@entry=0x56078934a880, hc=hc@entry=0x7ffd14c3beb0) at
../lib/netdev-linux.c:4804
#9 0x00005607879755da in htb_tc_install (details=0x56078934a880,
netdev=0x56078934c640) at ../lib/netdev-linux.c:4883
#10 htb_tc_install (netdev=0x56078934c640, details=0x56078934a880) at
../lib/netdev-linux.c:4875
#11 0x0000560787974937 in netdev_linux_set_qos (netdev_=0x56078934c640,
type=<optimized out>, details=0x56078934a880) at ../lib/netdev-linux.c:3054
#12 0x0000560787814ea5 in iface_configure_qos (qos=0x56078934a780,
iface=0x560789349fd0) at ../vswitchd/bridge.c:4845
#13 bridge_reconfigure (ovs_cfg=ovs_cfg@entry=0x5607892dbf90) at
../vswitchd/bridge.c:928
#14 0x0000560787817f7d in bridge_run () at ../vswitchd/bridge.c:3321
#15 0x000056078780d205 in main (argc=<optimized out>, argv=<optimized out>)
at ../vswitchd/ovs-vswitchd.c:130
A shell script to reproduce the issue is:
#!/bin/sh
apt-get install openvswitch-{common,switch}{,-dbgsym}
# Don't need it running on the system
systemctl stop openvswitch-switch
set -e
cleanup() {
ip link del veth0
rm /tmp/ovs/conf.db
}
trap cleanup EXIT
# Setup our environment
ip link add veth0 type veth peer veth1
mkdir -p /tmp/ovs
export OVS_RUNDIR=/tmp/ovs
export OVS_LOGDIR=/tmp/ovs
export OVS_DBDIR=/tmp/ovs
/usr/share/openvswitch/scripts/ovs-ctl start
ovs-vsctl add-br demo
ovs-vsctl add-port demo veth1
# Make it crash
ovs-vsctl set Port veth1 qos=@qos \
-- --id=@qos create QoS type=linux-htb queues:1=@high queues:2=@low \
-- --id=@high create Queue other-config:priority=1 other-config:min-rate=
0.1 \
-- --id=@low create Queue other-config:priority=6 other-config:min-rate=
0.05
We built the reproduction script based on speculation that
https://github.com/openvswitch/ovs/commit/6240c0b4c80ea3d8dd1bf77526b04b55742de2ce
is related to the crash. Notably we don't seem to run into the problem when
we pass in a specific maximum bandwidth instead of relying on the
interface's maximum bandwidth.
Sincerely,
Daryl Wang
_______________________________________________
discuss mailing list
[email protected]
https://mail.openvswitch.org/mailman/listinfo/ovs-discuss
