On 12/29/2012 07:43 PM, Ben Pfaff wrote: > On Sat, Dec 29, 2012 at 06:38:17PM -0600, Anand Krishnamurthy wrote: >> 2012-12-30T00:22:49Z|49582|dpif|WARN|system@ovs-system: failed to flow_del >> (No such file or directory) >> ipv4_tunnel(tun_id=0x0,src=172.19.0.42,dst=172.19.0.41,tos=0x0,ttl=64,flags()),in_port(5),eth(src=76:e8:f3:9f:95:a6,dst=01:80:c2:00:00:0e),eth_type(0x88cc) >> 2012-12-30T00:22:50Z|49583|ofproto_dpif|WARN|Dropped 3 log messages in last >> 45 seconds (most recently, 15 seconds ago) due to excessive rate >> 2012-12-30T00:22:50Z|49584|ofproto_dpif|WARN|unexpected flow on br0: >> ipv4_tunnel(tun_id=0x0,src=172.19.0.42,dst=172.19.0.41,tos=0x0,ttl=64,flags()),in_port(5),eth(src=76:e8:f3:9f:95:a6,dst=01:80:c2:00:00:0e),eth_type(0x88cc) > The above errors indicate a bug in Open vSwitch. Their form indicates > that you must be using a recent build from master. What commit are you > running?
We're using a build from commit a4454ac. I ran some other tests on the strange behavior we've been seeing with this build that might be related, deets below and in attachments. (tl;dr: ovswitches are failing to forward traffic from one GRE interface to another GRE interface correctly, but succeeds when a patch-pair is interposed between the tunnels.) ----- Experimental setup consists of thee hosts, named legolas, sam, and saruman. * Each host's LOM is assigned an IP in the 172.19.0.0/24 subnet, and they're all connected to a traditional flood-and-learn switch. * Each host runs an openvswitch, the internal port of which is assigned an IP address in the 172.20.0.0/16. * Each openvswitch has two GRE ports, each connecting to the other two openvswitches through the LOMs. (At this point, I have full connectivity across all the bridges, the GRE tunnels work as expected, and each host in 172.20.0.0 can ping each other.) * A slightly neutered Floodlight controller is connected to each ovswitch. Specifically, the Forwarding component of the controller has been removed. (At this point, as expected, I've lost all connectivity across all switches, as the controller doesn't install any flows by default.) This concludes the experimental setup. The procedure follows: 1. A flow was installed in the legolas switch, directing traffic from the local port out the GRE tunnel to sam, and vice-versa. 2. A flow was installed on the sam switch, directing traffic from legolas's GRE tunnel to saruman's GRE tunnel, and vice-versa. 3. A flow was installed on the saruman switch, directing traffic from sam's GRE tunnel to the local port, and vice-versa. (This is setup by attachment xperiment_1.sh.) My expectation would be that there should be connectivity between the bridge interfaces on legolas and saruman, and I should be able to see all that traffic entering and exiting the ovswitch at sam by packet-sniffing it's LOM. However, that is not the case. I see traffic destined for either machine at that interface, but it never subsequently departs. (see pcap1.txt) Weirdly, though, if I add a patch with both ends in the openvswitch on sam, and construct the rules such that the GRE traffic is forced through the patch cable prior to being directed to the other GRE tunnel, everything works again. (This is setup by xperiment_2.sh, and demonstrated by pcap2.txt.) Last odd thing: in the "un-patch-cabled" (broken) configuration, when I run ovs-appctl dpif/dump-flows br0 on sam, it displays these two flows, which I can't figure out: root@sam:~# sudo ovs-appctl dpif/dump-flows br0 ipv4_tunnel(tun_id=0x0,src=172.19.0.38,dst=172.19.0.43,tos=0x0,ttl=64,flags()),in_port(5),eth(src=16:93:be:ab:3d:4d,dst=5e:f6:c3:36:64:4b),eth_type(0x0800),ipv4(src=172.20.38.1,dst=172.20.44.1,proto=1,tos=0,ttl=64,frag=no),icmp(type=8,code=0), packets:1, bytes:98, used:1.412s, actions:set(ipv4_tunnel(tun_id=0x0,src=172.19.0.38,dst=172.19.0.43,tos=0x0,ttl=64,flags())),6 ipv4_tunnel(tun_id=0x0,src=172.19.0.44,dst=172.19.0.43,tos=0x0,ttl=64,flags()),in_port(6),eth(src=5e:f6:c3:36:64:4b,dst=16:93:be:ab:3d:4d),eth_type(0x0800),ipv4(src=172.20.44.1,dst=172.20.38.1,proto=1,tos=0,ttl=64,frag=no),icmp(type=8,code=0), packets:11, bytes:1078, used:1.068s, actions:set(ipv4_tunnel(tun_id=0x0,src=172.19.0.44,dst=172.19.0.43,tos=0x0,ttl=64,flags())),5 If I'm reading that correctly, it's not correctly re-encapsulating packets received on one GRE port and egressing via a different GRE port; it's just restoring the original GRE header instead. ??? -s
xperiment_1.sh
Description: application/shellscript
Script started on Sun 30 Dec 2012 12:37:46 AM CST
root@legolas:~# ovs-vsctl show
0df7239f-b2c4-4cf9-bb1f-72b04302a600
Bridge "br0"
Controller "tcp:127.0.0.1:6633"
is_connected: true
Port "br0"
Interface "br0"
type: internal
Port gre_sam
Interface gre_sam
type: gre
options: {remote_ip="172.19.0.43"}
Port gre_saruman
Interface gre_saruman
type: gre
options: {remote_ip="172.19.0.44"}
ovs_version: "1.9.90"
root@legolas:~# ovs-vsctl[K[K[K[K[Kofctl show br0
OFPT_FEATURES_REPLY (xid=0x2): dpid:00001693beab3d4d
n_tables:254, n_buffers:256
capabilities: FLOW_STATS TABLE_STATS PORT_STATS QUEUE_STATS ARP_MATCH_IP
actions: OUTPUT SET_VLAN_VID SET_VLAN_PCP STRIP_VLAN SET_DL_SRC SET_DL_DST
SET_NW_SRC SET_NW_DST SET_NW_TOS SET_TP_SRC SET_TP_DST ENQUEUE
2(gre_sam): addr:ae:01:1e:9e:d7:26
config: 0
state: 0
speed: 0 Mbps now, 0 Mbps max
3(gre_saruman): addr:36:cf:a6:ba:f0:46
config: 0
state: 0
speed: 0 Mbps now, 0 Mbps max
LOCAL(br0): addr:16:93:be:ab:3d:4d
config: 0
state: 0
speed: 0 Mbps now, 0 Mbps max
OFPT_GET_CONFIG_REPLY (xid=0x4): frags=normal miss_send_len=0
root@legolas:~# ovs[K[K[Kifconfig em2
em2 Link encap:Ethernet HWaddr 00:26:b9:8b:c3:da
inet addr:172.19.0.38 Bcast:172.19.0.255 Mask:255.255.255.0
inet6 addr: fe80::226:b9ff:fe8b:c3da/64 Scope:Link
UP BROADCAST RUNNING MULTICAST MTU:1500 Metric:1
RX packets:11703 errors:0 dropped:4719 overruns:0 frame:0
TX packets:28564 errors:0 dropped:0 overruns:0 carrier:0
collisions:0 txqueuelen:1000
RX bytes:1477037 (1.4 MB) TX bytes:2594547 (2.5 MB)
root@legolas:~# ifconfig br0
br0 Link encap:Ethernet HWaddr 16:93:be:ab:3d:4d
inet addr:172.20.38.1 Bcast:172.20.255.255 Mask:255.255.0.0
inet6 addr: fe80::c8e:5dff:fe98:5114/64 Scope:Link
UP BROADCAST RUNNING MTU:1500 Metric:1
RX packets:1232 errors:0 dropped:422 overruns:0 frame:0
TX packets:2273 errors:0 dropped:0 overruns:0 carrier:0
collisions:0 txqueuelen:0
RX bytes:126419 (126.4 KB) TX bytes:199101 (199.1 KB)
root@legolas:~# exit
exit
Script done on Sun 30 Dec 2012 12:38:16 AM CST
00:07:27.677016 IP 172.19.0.44 > 172.19.0.43: GREv0, length 102: IP 172.20.44.1 > 172.20.38.1: ICMP echo request, id 14418, seq 43, length 64 00:07:27.767904 IP 172.19.0.38 > 172.19.0.43: GREv0, length 102: IP 172.20.38.1 > 172.20.44.1: ICMP echo request, id 4946, seq 29, length 64 00:07:28.677023 IP 172.19.0.44 > 172.19.0.43: GREv0, length 102: IP 172.20.44.1 > 172.20.38.1: ICMP echo request, id 14418, seq 44, length 64 00:07:28.775891 IP 172.19.0.38 > 172.19.0.43: GREv0, length 102: IP 172.20.38.1 > 172.20.44.1: ICMP echo request, id 4946, seq 30, length 64 00:07:29.677035 IP 172.19.0.44 > 172.19.0.43: GREv0, length 102: IP 172.20.44.1 > 172.20.38.1: ICMP echo request, id 14418, seq 45, length 64 00:07:29.783878 IP 172.19.0.38 > 172.19.0.43: GREv0, length 102: IP 172.20.38.1 > 172.20.44.1: ICMP echo request, id 4946, seq 31, length 64 00:07:30.677041 IP 172.19.0.44 > 172.19.0.43: GREv0, length 102: IP 172.20.44.1 > 172.20.38.1: ICMP echo request, id 14418, seq 46, length 64 00:07:30.791865 IP 172.19.0.38 > 172.19.0.43: GREv0, length 102: IP 172.20.38.1 > 172.20.44.1: ICMP echo request, id 4946, seq 32, length 64 00:07:31.677051 IP 172.19.0.44 > 172.19.0.43: GREv0, length 102: IP 172.20.44.1 > 172.20.38.1: ICMP echo request, id 14418, seq 47, length 64 00:07:31.799849 IP 172.19.0.38 > 172.19.0.43: GREv0, length 102: IP 172.20.38.1 > 172.20.44.1: ICMP echo request, id 4946, seq 33, length 64 00:07:32.677060 IP 172.19.0.44 > 172.19.0.43: GREv0, length 102: IP 172.20.44.1 > 172.20.38.1: ICMP echo request, id 14418, seq 48, length 64 00:07:32.807844 IP 172.19.0.38 > 172.19.0.43: GREv0, length 102: IP 172.20.38.1 > 172.20.44.1: ICMP echo request, id 4946, seq 34, length 64 00:07:33.677071 IP 172.19.0.44 > 172.19.0.43: GREv0, length 102: IP 172.20.44.1 > 172.20.38.1: ICMP echo request, id 14418, seq 49, length 64 00:07:33.815828 IP 172.19.0.38 > 172.19.0.43: GREv0, length 102: IP 172.20.38.1 > 172.20.44.1: ICMP echo request, id 4946, seq 35, length 64 00:07:34.677078 IP 172.19.0.44 > 172.19.0.43: GREv0, length 102: IP 172.20.44.1 > 172.20.38.1: ICMP echo request, id 14418, seq 50, length 64 00:07:34.823814 IP 172.19.0.38 > 172.19.0.43: GREv0, length 102: IP 172.20.38.1 > 172.20.44.1: ICMP echo request, id 4946, seq 36, length 64 00:07:35.677087 IP 172.19.0.44 > 172.19.0.43: GREv0, length 102: IP 172.20.44.1 > 172.20.38.1: ICMP echo request, id 14418, seq 51, length 64 00:07:35.831804 IP 172.19.0.38 > 172.19.0.43: GREv0, length 102: IP 172.20.38.1 > 172.20.44.1: ICMP echo request, id 4946, seq 37, length 64 00:07:36.677097 IP 172.19.0.44 > 172.19.0.43: GREv0, length 102: IP 172.20.44.1 > 172.20.38.1: ICMP echo request, id 14418, seq 52, length 64 00:07:36.839790 IP 172.19.0.38 > 172.19.0.43: GREv0, length 102: IP 172.20.38.1 > 172.20.44.1: ICMP echo request, id 4946, seq 38, length 64
00:18:50.687226 IP 172.19.0.44 > 172.19.0.43: GREv0, length 102: IP 172.20.44.1 > 172.20.38.1: ICMP echo request, id 14418, seq 726, length 64 00:18:50.688550 IP 172.19.0.43 > 172.19.0.38: GREv0, length 102: IP 172.20.44.1 > 172.20.38.1: ICMP echo request, id 14418, seq 726, length 64 00:18:50.688644 IP 172.19.0.38 > 172.19.0.43: GREv0, length 102: IP 172.20.38.1 > 172.20.44.1: ICMP echo reply, id 14418, seq 726, length 64 00:18:50.688663 IP 172.19.0.43 > 172.19.0.44: GREv0, length 102: IP 172.20.38.1 > 172.20.44.1: ICMP echo reply, id 14418, seq 726, length 64 00:18:51.185365 IP 172.19.0.38 > 172.19.0.43: GREv0, length 102: IP 172.20.38.1 > 172.20.44.1: ICMP echo request, id 4946, seq 709, length 64 00:18:51.688984 IP 172.19.0.44 > 172.19.0.43: GREv0, length 102: IP 172.20.44.1 > 172.20.38.1: ICMP echo request, id 14418, seq 727, length 64 00:18:51.689306 IP 172.19.0.43 > 172.19.0.38: GREv0, length 102: IP 172.20.44.1 > 172.20.38.1: ICMP echo request, id 14418, seq 727, length 64 00:18:51.689412 IP 172.19.0.38 > 172.19.0.43: GREv0, length 102: IP 172.20.38.1 > 172.20.44.1: ICMP echo reply, id 14418, seq 727, length 64 00:18:51.689434 IP 172.19.0.43 > 172.19.0.44: GREv0, length 102: IP 172.20.38.1 > 172.20.44.1: ICMP echo reply, id 14418, seq 727, length 64 00:18:52.185352 IP 172.19.0.38 > 172.19.0.43: GREv0, length 102: IP 172.20.38.1 > 172.20.44.1: ICMP echo request, id 4946, seq 710, length 64
Script started on Sun 30 Dec 2012 12:35:32 AM CST
root@sam:~# ovs-vsctl show
br[K[K[K[K[K[K[K[K[K[K[K[K[Kofctl show br0
OFPT_FEATURES_REPLY (xid=0x2): dpid:0000eac5d9b77640
n_tables:254, n_buffers:256
capabilities: FLOW_STATS TABLE_STATS PORT_STATS QUEUE_STATS ARP_MATCH_IP
actions: OUTPUT SET_VLAN_VID SET_VLAN_PCP STRIP_VLAN SET_DL_SRC SET_DL_DST
SET_NW_SRC SET_NW_DST SET_NW_TOS SET_TP_SRC SET_TP_DST ENQUEUE
1(patch0): addr:92:ab:f9:8e:3f:f6
config: 0
state: 0
speed: 0 Mbps now, 0 Mbps max
2(patch1): addr:92:37:f2:30:d0:f3
config: 0
state: 0
speed: 0 Mbps now, 0 Mbps max
4(gre_legolas): addr:9e:07:a5:f8:ca:5c
config: 0
state: 0
speed: 0 Mbps now, 0 Mbps max
6(gre_saruman): addr:82:9b:69:79:0f:ab
config: 0
state: 0
speed: 0 Mbps now, 0 Mbps max
LOCAL(br0): addr:ea:c5:d9:b7:76:40
config: 0
state: 0
speed: 0 Mbps now, 0 Mbps max
OFPT_GET_CONFIG_REPLY (xid=0x4): frags=normal miss_send_len=0
root@sam:~# ovs-vsctl show
b51980ec-999c-45ac-b928-35eab23d68b4
Bridge "br0"
Controller "tcp:127.0.0.1:6633"
is_connected: true
Port gre_legolas
Interface gre_legolas
type: gre
options: {remote_ip="172.19.0.38"}
Port gre_saruman
Interface gre_saruman
type: gre
options: {remote_ip="172.19.0.44"}
Port "patch0"
Interface "patch0"
type: patch
options: {peer="patch1"}
Port "patch1"
Interface "patch1"
type: patch
options: {peer="patch0"}
Port "br0"
Interface "br0"
type: internal
ovs_version: "1.9.90"
root@sam:~# ifconfig em2
em2 Link encap:Ethernet HWaddr 00:26:b9:8b:c9:04
inet addr:172.19.0.43 Bcast:172.19.0.255 Mask:255.255.255.0
inet6 addr: fe80::226:b9ff:fe8b:c904/64 Scope:Link
UP BROADCAST RUNNING MULTICAST MTU:1500 Metric:1
RX packets:26474 errors:0 dropped:14378 overruns:0 frame:0
TX packets:29985 errors:0 dropped:0 overruns:0 carrier:0
collisions:0 txqueuelen:1000
RX bytes:3171425 (3.1 MB) TX bytes:2627980 (2.6 MB)
root@sam:~# ifconfig br0
br0 Link encap:Ethernet HWaddr ea:c5:d9:b7:76:40
inet addr:172.20.43.1 Bcast:172.20.255.255 Mask:255.255.0.0
inet6 addr: fe80::e40e:10ff:fe89:df47/64 Scope:Link
UP BROADCAST RUNNING MTU:1500 Metric:1
RX packets:56116 errors:0 dropped:34316 overruns:0 frame:0
TX packets:28283 errors:0 dropped:0 overruns:0 carrier:0
collisions:0 txqueuelen:0
RX bytes:99833215 (99.8 MB) TX bytes:195321649 (195.3 MB)
root@sam:~# exit
exit
Script done on Sun 30 Dec 2012 12:36:16 AM CST
Script started on Sun 30 Dec 2012 12:36:50 AM CST
root@saruman:~# ovs-vsctl show
1acd0195-6a31-4739-bbbe-1c8b21ed924f
Bridge "br0"
Controller "tcp:127.0.0.1:6633"
is_connected: true
Port gre_sam
Interface gre_sam
type: gre
options: {remote_ip="172.19.0.43"}
Port "br0"
Interface "br0"
type: internal
Port gre_legolas
Interface gre_legolas
type: gre
options: {remote_ip="172.19.0.38"}
ovs_version: "1.9.90"
root@saruman:~# ovs-ofctl show br0
OFPT_FEATURES_REPLY (xid=0x2): dpid:00005ef6c336644b
n_tables:254, n_buffers:256
capabilities: FLOW_STATS TABLE_STATS PORT_STATS QUEUE_STATS ARP_MATCH_IP
actions: OUTPUT SET_VLAN_VID SET_VLAN_PCP STRIP_VLAN SET_DL_SRC SET_DL_DST
SET_NW_SRC SET_NW_DST SET_NW_TOS SET_TP_SRC SET_TP_DST ENQUEUE
12(gre_legolas): addr:22:19:7e:b7:20:92
config: 0
state: 0
speed: 0 Mbps now, 0 Mbps max
14(gre_sam): addr:1a:e2:9e:6c:9f:97
config: 0
state: 0
speed: 0 Mbps now, 0 Mbps max
LOCAL(br0): addr:5e:f6:c3:36:64:4b
config: 0
state: 0
speed: 0 Mbps now, 0 Mbps max
OFPT_GET_CONFIG_REPLY (xid=0x4): frags=normal miss_send_len=0
root@saruman:~# ifconfig em2
em2 Link encap:Ethernet HWaddr 00:26:b9:8b:c9:34
inet addr:172.19.0.44 Bcast:172.19.0.255 Mask:255.255.255.0
inet6 addr: fe80::226:b9ff:fe8b:c934/64 Scope:Link
UP BROADCAST RUNNING MULTICAST MTU:1500 Metric:1
RX packets:7332875 errors:0 dropped:12951 overruns:0 frame:0
TX packets:269465 errors:0 dropped:0 overruns:0 carrier:0
collisions:0 txqueuelen:1000
RX bytes:5770921351 (5.7 GB) TX bytes:28518947 (28.5 MB)
root@saruman:~# ifconfig br0
br0 Link encap:Ethernet HWaddr 5e:f6:c3:36:64:4b
inet addr:172.20.44.1 Bcast:172.20.255.255 Mask:255.255.0.0
inet6 addr: fe80::d0e9:7eff:fe14:b0e0/64 Scope:Link
UP BROADCAST RUNNING MTU:1500 Metric:1
RX packets:9500 errors:0 dropped:8129 overruns:0 frame:0
TX packets:1455 errors:0 dropped:0 overruns:0 carrier:0
collisions:0 txqueuelen:0
RX bytes:610160 (610.1 KB) TX bytes:199452 (199.4 KB)
root@saruman:~# exit
exit
Script done on Sun 30 Dec 2012 12:37:16 AM CST
xperiment_2.sh
Description: application/shellscript
_______________________________________________ discuss mailing list discuss@openvswitch.org http://openvswitch.org/mailman/listinfo/discuss
