Folks,
I have a small testbed with 4 ubuntu servers, fully meshed with VXLAN. To avoid
loops I use Spanning-Tree.
The 4 servers run Ubuntu 14.10 fully updated.
I upgraded manually to openvswitch 2.3.1 with the vivid packages from ubuntu.
3 Servers run kvm and consume via rbd/ceph images. CEPH runs over openvswitch
to sync data.
The 3 KVM servers share via VXLAN flat networks trunked with VLAN's.
These physical servers are single homed and therefore I need to bridge eth0
also to the outside world.
Randomly these servers loose connectivity to the outside world (eth0).
Troubleshooting shows, that it is often related to STP events (VM's
restarting/shutdown/hosts booting, etc).
After very long searches I think I found the culprit: A lock file
".conf.db.~lock~" is present in /etc/openvswitch directory. If I delete that
file and reboot the host connectivity reappears.
When the host looses connectivity, no ipv4 traffic is forwarded anymore. No
real culprits are visible in ovs-vswitch logs, except that a STP topology
change happened.
local ovsbr0p1 bridged to eth0 interface can be ipv4 pinged. Gateway cannot be
pinged anymore (sometimes ipv6 connectivity still works)
Any tips how to troubleshoot this and get rid of this sudden loss of
connectivity ? Same problem happened on OVS 2.0 release.
Second question: How can I get rid of SPT events when VM's boot or similar ? On
HW switches there is a concept of Access-Ports for end devices, where no STP
event will be flood to the fabric.
thanks
marco
Below some outputs that could help in troubleshooting:
root@nuv-vir-kvm-server-3 ~ # ovs-vsctl show
5733eb7c-f4ac-446c-bfd1-c0121c0337b7
Bridge "ovsbr0"
Port "eth0"
Interface "eth0"
Port "ovsbr0p1"
Interface "ovsbr0p1"
type: internal
Port "ovsbr0"
Interface "ovsbr0"
type: internal
Bridge "ovsbr1"
Port "ovsbr1.8p1"
tag: 8
Interface "ovsbr1.8p1"
type: internal
Port "ovsbr1.12p1"
tag: 12
Interface "ovsbr1.12p1"
type: internal
Port "ovsbr1.3p1"
tag: 3
Interface "ovsbr1.3p1"
type: internal
Port "ovsbr1.4"
tag: 4
Interface "ovsbr1.4"
type: internal
Port "ovsbr1.101p1"
tag: 101
Interface "ovsbr1.101p1"
type: internal
Port "ovsvx5"
trunks: [1, 2, 3, 4, 5, 6, 7, 8, 9, 10, 11, 12, 20, 30, 101]
Interface "ovsvx5"
type: vxlan
options: {key=flow, remote_ip="46.4.100.54"}
Port "ovsbr1.1"
tag: 1
Interface "ovsbr1.1"
type: internal
Port "ovsbr1.9"
tag: 9
Interface "ovsbr1.9"
type: internal
Port "ovsbr1.2"
tag: 2
Interface "ovsbr1.2"
type: internal
Port "ovsbr1.20"
tag: 20
Interface "ovsbr1.20"
type: internal
Port "ovsbr1.30p1"
tag: 30
Interface "ovsbr1.30p1"
type: internal
Port "ovsbr1.101"
tag: 101
Interface "ovsbr1.101"
type: internal
Port "ovsbr1.5"
tag: 5
Interface "ovsbr1.5"
type: internal
Port "ovsbr1.7"
tag: 7
Interface "ovsbr1.7"
type: internal
Port "ovsbr1.1p1"
tag: 1
Interface "ovsbr1.1p1"
type: internal
Port "ovsbr1.6"
tag: 6
Interface "ovsbr1.6"
type: internal
Port "ovsbr1.11p1"
tag: 11
Interface "ovsbr1.11p1"
type: internal
Port "ovsbr1.4p1"
tag: 4
Interface "ovsbr1.4p1"
type: internal
Port "ovsbr1"
Interface "ovsbr1"
type: internal
Port "ovsbr1.8"
tag: 8
Interface "ovsbr1.8"
type: internal
Port "ovsbr1.12"
tag: 12
Interface "ovsbr1.12"
type: internal
Port "ovsbr1.11"
tag: 11
Interface "ovsbr1.11"
type: internal
Port "ovsbr1.10p1"
tag: 10
Interface "ovsbr1.10p1"
type: internal
Port "ovsbr1.7p1"
tag: 7
Interface "ovsbr1.7p1"
type: internal
Port "ovsbr1.2p1"
tag: 2
Interface "ovsbr1.2p1"
type: internal
Port "ovsbr1.9p1"
tag: 9
Interface "ovsbr1.9p1"
type: internal
Port "ovsbr1.30"
tag: 30
Interface "ovsbr1.30"
type: internal
Port "ovsvxnuv-vir-aci-ubuntu-1"
trunks: [1, 2, 3, 4, 5, 6, 7, 8, 9, 10, 11, 12, 20, 30, 101]
Interface "ovsvxnuv-vir-aci-ubuntu-1"
type: vxlan
options: {key=flow, remote_ip="46.4.17.109"}
Port "ovsbr1.10"
tag: 10
Interface "ovsbr1.10"
type: internal
Port "ovsvx7"
trunks: [1, 2, 3, 4, 5, 6, 7, 8, 9, 10, 11, 12, 20, 30, 101]
Interface "ovsvx7"
type: vxlan
options: {key=flow, remote_ip="46.4.17.67"}
Port "ovsbr1.5p1"
tag: 5
Interface "ovsbr1.5p1"
type: internal
Port "ovsvx6"
trunks: [1, 2, 3, 4, 5, 6, 7, 8, 9, 10, 11, 12, 20, 30, 101]
Interface "ovsvx6"
type: vxlan
options: {key=flow, remote_ip="46.4.109.146"}
Port "ovsbr1.6p1"
tag: 6
Interface "ovsbr1.6p1"
type: internal
Port "ovsbr1.3"
tag: 3
Interface "ovsbr1.3"
type: internal
Port "ovsbr1.20p1"
tag: 20
Interface "ovsbr1.20p1"
type: internal
ovs_version: "2.3.1"
_uuid : c960a8da-aa1a-4c8a-a3f3-1b7863424313
bond_active_slave : []
bond_downdelay : 0
bond_fake_iface : false
bond_mode : []
bond_updelay : 0
external_ids : {}
fake_bridge : false
interfaces : [e8ec16b3-1a83-44a0-b3f8-f3dccfdde91d]
lacp : []
mac : []
name : "eth0"
other_config : {}
qos : []
statistics : {}
status : {}
tag : []
trunks : []
vlan_mode : []
root@nuv-vir-kvm-server-5 ~ # ovs-vsctl list port
_uuid : 6b14b501-a85b-4fc0-8cd3-1302830ff35c
bond_active_slave : []
bond_downdelay : 0
bond_fake_iface : false
bond_mode : []
bond_updelay : 0
external_ids : {}
fake_bridge : false
interfaces : [7dfc6aa1-23bf-4784-98c3-583ea20884c6]
lacp : []
mac : []
name : "ovsbr1.1p1"
other_config : {}
qos : []
statistics : {}
status : {}
tag : 1
trunks : []
vlan_mode : []
_uuid : eec4c406-3cf5-4507-9ca3-251741631e77
bond_active_slave : []
bond_downdelay : 0
bond_fake_iface : false
bond_mode : []
bond_updelay : 0
external_ids : {}
fake_bridge : false
interfaces : [8e88f0c9-31e0-4c44-81b8-53511ab8d8bd]
lacp : []
mac : []
name : "ovsbr1"
other_config : {}
qos : []
statistics : {}
status : {}
tag : []
trunks : []
vlan_mode : []
_uuid : a191cc12-89d3-4a79-bdaf-55502c5b213d
bond_active_slave : []
bond_downdelay : 0
bond_fake_iface : false
bond_mode : []
bond_updelay : 0
external_ids : {}
fake_bridge : false
interfaces : [4281866e-11b4-4c04-8a6c-066aea7af246]
lacp : []
mac : []
name : "ovsbr0p1"
other_config : {}
qos : []
statistics : {}
status : {}
tag : []
trunks : []
vlan_mode : []
_uuid : 22ee48dc-7fec-4742-b577-a5293f7e3bd9
admin_state : up
bfd : {}
bfd_status : {}
cfm_fault : []
cfm_fault_status : []
cfm_flap_count : []
cfm_health : []
cfm_mpid : []
cfm_remote_mpids : []
cfm_remote_opstate : []
duplex : []
external_ids : {}
ifindex : 5
ingress_policing_burst: 0
ingress_policing_rate: 0
lacp_current : []
link_resets : 1
link_speed : []
link_state : up
mac : []
mac_in_use : "c8:60:00:5f:f0:3f"
mtu : 1500
name : "ovsbr0"
ofport : 65534
ofport_request : []
options : {}
other_config : {}
statistics : {collisions=0, rx_bytes=360, rx_crc_err=0, rx_dropped=0,
rx_errors=0, rx_frame_err=0, rx_over_err=0, rx_packets=4, tx_bytes=44058,
tx_dropped=0, tx_errors=0, tx_packets=328}
status : {driver_name=openvswitch}
type : internal
_uuid : e8ec16b3-1a83-44a0-b3f8-f3dccfdde91d
admin_state : up
bfd : {}
bfd_status : {}
cfm_fault : []
cfm_fault_status : []
cfm_flap_count : []
cfm_health : []
cfm_mpid : []
cfm_remote_mpids : []
cfm_remote_opstate : []
duplex : full
external_ids : {}
ifindex : 2
ingress_policing_burst: 0
ingress_policing_rate: 0
lacp_current : []
link_resets : 1
link_speed : 1000000000
link_state : up
mac : []
mac_in_use : "c8:60:00:5f:f0:3f"
mtu : 1500
name : "eth0"
ofport : 2
ofport_request : []
options : {}
other_config : {}
statistics : {collisions=0, rx_bytes=1381410080, rx_crc_err=0,
rx_dropped=0, rx_errors=0, rx_frame_err=0, rx_over_err=0, rx_packets=6910253,
tx_bytes=1055169046, tx_dropped=0, tx_errors=0, tx_packets=2504716}
status : {driver_name="r8169", driver_version="2.3LK-NAPI",
firmware_version="rtl_nic/rtl8168e-2.fw"}
type : ""
_uuid : 4281866e-11b4-4c04-8a6c-066aea7af246
admin_state : up
bfd : {}
bfd_status : {}
cfm_fault : []
cfm_fault_status : []
cfm_flap_count : []
cfm_health : []
cfm_mpid : []
cfm_remote_mpids : []
cfm_remote_opstate : []
duplex : []
external_ids : {}
ifindex : 4
ingress_policing_burst: 0
ingress_policing_rate: 0
lacp_current : []
link_resets : 1
link_speed : []
link_state : up
mac : []
mac_in_use : "62:ee:a5:83:b5:7e"
mtu : 1500
name : "ovsbr0p1"
ofport : 1
ofport_request : []
options : {}
other_config : {}
statistics : {collisions=0, rx_bytes=1053520420, rx_crc_err=0,
rx_dropped=0, rx_errors=0, rx_frame_err=0, rx_over_err=0, rx_packets=2474191,
tx_bytes=9971342322, tx_dropped=0, tx_errors=0, tx_packets=6910215}
status : {driver_name=openvswitch}
type : internal
_uuid : 7dfc6aa1-23bf-4784-98c3-583ea20884c6
admin_state : up
bfd : {}
bfd_status : {}
cfm_fault : []
cfm_fault_status : []
cfm_flap_count : []
cfm_health : []
cfm_mpid : []
cfm_remote_mpids : []
cfm_remote_opstate : []
duplex : []
external_ids : {}
ifindex : 36
ingress_policing_burst: 0
ingress_policing_rate: 0
lacp_current : []
link_resets : 1
link_speed : []
link_state : up
mac : []
mac_in_use : "6e:e8:ec:44:f9:1a"
mtu : 1400
name : "ovsbr1.1p1"
ofport : 2
ofport_request : []
options : {}
other_config : {}
statistics : {collisions=0, rx_bytes=800790181, rx_crc_err=0,
rx_dropped=0, rx_errors=0, rx_frame_err=0, rx_over_err=0, rx_packets=2044691,
tx_bytes=9595662921, tx_dropped=0, tx_errors=0, tx_packets=6887123}
status : {driver_name=openvswitch}
type : internal
_uuid : 8c602077-b2b1-4f2c-9201-93304008a814
bond_active_slave : []
bond_downdelay : 0
bond_fake_iface : false
bond_mode : []
bond_updelay : 0
external_ids : {}
fake_bridge : false
interfaces : [18cb6a6c-1f66-4eb5-bc78-304e8b404ac0]
lacp : []
mac : []
name : "ovsvx3"
other_config : {}
qos : []
statistics : {stp_error_count=0, stp_rx_count=311, stp_tx_count=4}
status : {stp_port_id="8002", stp_role=root,
stp_sec_in_state="592", stp_state=forwarding}
tag : []
trunks : [1, 2, 3, 4, 5, 6, 7, 8, 9, 10, 11, 12, 20, 30, 101]
vlan_mode : trunk
2015-02-05T10:54:04.316Z|00082|bridge|INFO|bridge ovsbr1: added interface
ovsvxnuv-vir-aci-ubuntu-1 on port 35
2015-02-05T10:54:34.562Z|00083|stp|INFO|ovsbr1: detected topology change.
2015-02-05T18:13:52.953Z|00014|stp(handler10)|INFO|ovsbr1: detected topology
change.
2015-02-05T20:58:13.680Z|00084|stp|INFO|ovsbr1: detected topology change.
2015-02-05T20:58:14.181Z|00085|stp|INFO|ovsbr1: detected topology change.
2015-02-05T20:59:49.410Z|00086|stp|INFO|ovsbr1: detected topology change.
2015-02-05T21:13:56.899Z|00015|stp(handler10)|INFO|ovsbr1: detected topology
change.
2015-02-05T21:16:05.136Z|00087|stp|INFO|ovsbr1: detected topology change.
2015-02-05T21:16:05.637Z|00088|stp|INFO|ovsbr1: detected topology change.
2015-02-05T21:16:34.178Z|00089|stp|INFO|ovsbr1: detected topology change.
2015-02-05T21:39:46.234Z|00016|stp(handler10)|INFO|ovsbr1: detected topology
change.
2015-02-05T22:44:19.686Z|00090|stp|INFO|ovsbr1: detected topology change.
2015-02-05T22:44:49.218Z|00091|stp|INFO|ovsbr1: detected topology change.
2015-02-05T23:01:55.688Z|00001|fatal_signal(urcu5)|WARN|terminating with signal
15 (Terminated)
2015-02-05T23:07:39.407Z|00001|vlog|INFO|opened log file
/var/log/openvswitch/ovs-vswitchd.log
2015-02-05T23:07:39.446Z|00002|reconnect|INFO|unix:/var/run/openvswitch/db.sock:
connecting...
2015-02-05T23:07:39.446Z|00003|reconnect|INFO|unix:/var/run/openvswitch/db.sock:
connected
2015-02-05T23:07:39.450Z|00004|dpif|WARN|system@ovs-system: failed to
put[create] (Invalid argument)
skb_priority(0),skb_mark(0),recirc_id(0x1),dp_hash(0x1),in_port(0),eth(src=00:00:00:00:00:00,dst=00:00:00:00:00:00)
2015-02-05T23:07:39.450Z|00005|ofproto_dpif|INFO|system@ovs-system: Datapath
does not support recirculation
2015-02-05T23:07:39.450Z|00006|dpif|WARN|system@ovs-system: execute
userspace(pid=0,userdata(00000000)) failed (Invalid argument) on packet
metadata=0,in_port=0,vlan_tci=0x0000,dl_src=00:00:00:00:00:00,dl_dst=00:00:00:00:00:00,dl_type=0x1234
2015-02-05T23:07:39.450Z|00007|ofproto_dpif|WARN|system@ovs-system:
variable-length userdata feature probe failed (Invalid argument)
2015-02-05T23:07:39.450Z|00008|dpif|WARN|system@ovs-system: failed to
put[create] (Invalid argument)
skb_priority(0),skb_mark(0),in_port(0),eth(src=00:00:00:00:00:00,dst=00:00:00:00:00:00),eth_type(0x8847),mpls(label=0,tc=0,ttl=0,bos=1)
2015-02-05T23:07:39.450Z|00009|ofproto_dpif|INFO|system@ovs-system: MPLS label
stack length probed as 0
2015-02-05T23:07:39.454Z|00010|dpif|WARN|system@ovs-system: failed to query
port ovsvxnuv-vir-aci-ubuntu-1: Invalid argument
2015-02-05T23:07:39.455Z|00011|bridge|INFO|bridge ovsbr0: added interface
ovsbr0p1 on port 1
2015-02-05T23:07:39.456Z|00012|bridge|INFO|bridge ovsbr0: added interface eth0
on port 2
2015-02-05T23:07:39.457Z|00013|bridge|INFO|bridge ovsbr0: added interface
ovsbr0 on port 65534
2015-02-05T23:07:39.459Z|00014|bridge|INFO|bridge ovsbr1: added interface
ovsbr1.2p1 on port 4
2015-02-05T23:07:39.460Z|00015|bridge|INFO|bridge ovsbr1: added interface
ovsbr1.6 on port 11
2015-02-05T23:07:39.461Z|00016|bridge|INFO|bridge ovsbr1: added interface
ovsvx6 on port 31
2015-02-05T23:07:39.463Z|00017|bridge|INFO|bridge ovsbr1: added interface
ovsbr1.7 on port 13
2015-02-05T23:07:39.467Z|00018|bridge|INFO|bridge ovsbr1: added interface
ovsbr1.9 on port 17
2015-02-05T23:07:39.469Z|00019|bridge|INFO|bridge ovsbr1: added interface
ovsbr1.9p1 on port 18
2015-02-05T23:07:39.472Z|00020|bridge|INFO|bridge ovsbr1: added interface
ovsbr1.1 on port 1
2015-02-05T23:07:39.473Z|00021|bridge|INFO|bridge ovsbr1: added interface
ovsbr1.6p1 on port 12
2015-02-05T23:07:39.476Z|00022|bridge|INFO|bridge ovsbr1: added interface
ovsbr1.10p1 on port 20
2015-02-05T23:07:39.476Z|00023|bridge|INFO|bridge ovsbr1: added interface
ovsvx3 on port 33
2015-02-05T23:07:39.481Z|00024|bridge|INFO|bridge ovsbr1: added interface
ovsbr1.4p1 on port 8
2015-02-05T23:07:39.485Z|00025|bridge|INFO|bridge ovsbr1: added interface
ovsbr1 on port 65534
2015-02-05T23:07:39.489Z|00001|ofproto_dpif_upcall(handler1)|INFO|received
packet on unassociated datapath port 1
2015-02-05T23:07:39.490Z|00026|bridge|INFO|bridge ovsbr1: added interface
ovsbr1.3p1 on port 6
2015-02-05T23:07:39.495Z|00027|bridge|INFO|bridge ovsbr1: added interface
ovsbr1.10 on port 19
2015-02-05T23:07:39.497Z|00002|ofproto_dpif_upcall(handler1)|INFO|received
packet on unassociated datapath port 3
2015-02-05T23:07:39.508Z|00028|bridge|INFO|bridge ovsbr1: added interface
ovsbr1.30 on port 27
2015-02-05T23:07:39.510Z|00029|bridge|INFO|bridge ovsbr1: added interface
ovsbr1.7p1 on port 14
2015-02-05T23:07:39.514Z|00030|bridge|INFO|bridge ovsbr1: added interface
ovsbr1.11p1 on port 22
2015-02-05T23:07:39.514Z|00031|bridge|INFO|bridge ovsbr1: added interface
ovsvx7 on port 34
2015-02-05T23:07:39.516Z|00032|bridge|INFO|bridge ovsbr1: added interface
ovsbr1.20p1 on port 26
2015-02-05T23:07:39.516Z|00033|bridge|INFO|bridge ovsbr1: added interface
ovsvxnuv-vir-aci-ubuntu-1 on port 35
2015-02-05T23:07:39.518Z|00034|bridge|INFO|bridge ovsbr1: added interface
ovsbr1.20 on port 25
2015-02-05T23:07:39.521Z|00035|bridge|INFO|bridge ovsbr1: added interface
ovsbr1.3 on port 5
2015-02-05T23:07:39.525Z|00036|bridge|INFO|bridge ovsbr1: added interface
ovsbr1.4 on port 7
2015-02-05T23:07:39.529Z|00037|bridge|INFO|bridge ovsbr1: added interface
ovsbr1.5p1 on port 10
2015-02-05T23:07:39.532Z|00038|bridge|INFO|bridge ovsbr1: added interface
ovsbr1.8p1 on port 16
2015-02-05T23:07:39.539Z|00003|ofproto_dpif_upcall(handler1)|INFO|received
packet on unassociated datapath port 10
2015-02-05T23:07:39.541Z|00039|bridge|INFO|bridge ovsbr1: added interface
ovsbr1.11 on port 21
2015-02-05T23:07:39.546Z|00040|bridge|INFO|bridge ovsbr1: added interface
ovsbr1.5 on port 9
2015-02-05T23:07:39.549Z|00041|bridge|INFO|bridge ovsbr1: added interface
ovsbr1.101 on port 29
2015-02-05T23:07:39.553Z|00042|bridge|INFO|bridge ovsbr1: added interface
ovsbr1.2 on port 3
2015-02-05T23:07:39.563Z|00043|bridge|INFO|bridge ovsbr1: added interface
ovsbr1.12p1 on port 24
2015-02-05T23:07:39.569Z|00044|bridge|INFO|bridge ovsbr1: added interface
ovsbr1.8 on port 15
2015-02-05T23:07:39.570Z|00004|ofproto_dpif_upcall(handler1)|INFO|received
packet on unassociated datapath port 14
2015-02-05T23:07:39.573Z|00045|bridge|INFO|bridge ovsbr1: added interface
ovsbr1.12 on port 23
2015-02-05T23:07:39.575Z|00046|bridge|INFO|bridge ovsbr1: added interface
ovsbr1.101p1 on port 30
2015-02-05T23:07:39.581Z|00047|bridge|INFO|bridge ovsbr1: added interface
ovsbr1.30p1 on port 28
2015-02-05T23:07:39.584Z|00048|bridge|INFO|bridge ovsbr1: added interface
ovsbr1.1p1 on port 2
2015-02-05T23:07:39.584Z|00049|bridge|INFO|bridge ovsbr0: using datapath ID
0000c860005ff03f
2015-02-05T23:07:39.584Z|00050|connmgr|INFO|ovsbr0: added service controller
"punix:/var/run/openvswitch/ovsbr0.mgmt"
2015-02-05T23:07:39.584Z|00051|bridge|INFO|bridge ovsbr1: using datapath ID
0000de63bff5aa47
2015-02-05T23:07:39.585Z|00052|connmgr|INFO|ovsbr1: added service controller
"punix:/var/run/openvswitch/ovsbr1.mgmt"
2015-02-05T23:07:39.586Z|00053|dpif|WARN|system@ovs-system: failed to query
port ovsvxnuv-vir-aci-ubuntu-1: Invalid argument
2015-02-05T23:07:39.595Z|00054|bridge|INFO|ovs-vswitchd (Open vSwitch) 2.3.1
2015-02-05T23:07:49.598Z|00055|memory|INFO|49472 kB peak resident set size
after 10.2 seconds
2015-02-05T23:07:49.598Z|00056|memory|INFO|handlers:2 ports:38 revalidators:2
rules:12 udpif keys:11
2015-02-05T23:08:09.607Z|00057|stp|INFO|ovsbr1: detected topology change.
2015-02-05T23:08:09.607Z|00058|stp|INFO|ovsbr1: detected topology change.
2015-02-05T23:08:09.607Z|00059|stp|INFO|ovsbr1: detected topology change.
2015-02-05T23:08:45.990Z|00060|bridge|INFO|bridge ovsbr1: added interface vnet0
on port 32
2015-02-05T23:09:16.213Z|00061|stp|INFO|ovsbr1: detected topology change.
_______________________________________________
discuss mailing list
discuss@openvswitch.org
http://openvswitch.org/mailman/listinfo/discuss
