Hello,
I'm using OpenStack icehouse and OpenvSwitch 2.1.3
The problem is that packets from VM's can go out, but never return to VM's.
#
# Cluster setup is as follows:
#
Compute node 2:
Vxlan: 192.168.100.2
Compute node 4:
Vxlan: 192.168.100.4
Network node:
Vxlan: 192.168.100.1
External IP: 10.61.4.194
Gateway IP (physical router): 10.61.4.193
#
# Compute node 2:
#
VM IP: 192.168.102.18/24
FloatingIP : 10.61.4.195
All data below is collected while running "ping 8.8.8.8" from VM
(192.168.102.18).
As you can see icmp packet is successfully sent to 8.8.8.8 (192.168.102.18
-> 10.61.4.195 -> 8.8.8.8)
But the packet is only returning to external IP and never pass that point
(8.8.8.8 -> 10.61.4.195)
I'm either missing something really stupid or there is a defect somewhere.
BR-INT and BR-EX bridges are connected by int-br-ex and phy-br-ex ports
(patch):
BR-INT <-> int-br-ex <-> int-br-phy <-> BR-EX
Both ends (VM and external) are on the same (qrouter) namespace, but
different bridges.
Because from the qrouter namespace I can reach both ends (VM and external) I
think problem is somewhere in transferring packet from phy-br-ex (br-ex) to
int-br-ex (br-int).
Can anyone take a look? It's driving me crazy as I can't find whats wrong
for a long. Any help on tracing the root cause would be highly appreciated.
#
# Network node
# Namespace: qrouter-5078354f-0214-455d-bb4d-282b6d98fb86
# QR- is on br-int tag:1, QG- is on br-ex no tag
#
400: qr-aa72cd87-16: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc
noqueue state UNKNOWN
link/ether fa:16:3e:29:a5:54 brd ff:ff:ff:ff:ff:ff
inet 192.168.102.1/24 brd 192.168.102.255 scope global qr-aa72cd87-16
401: qg-d6beba08-56: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc
noqueue state UNKNOWN
link/ether fa:16:3e:26:a7:52 brd ff:ff:ff:ff:ff:ff
inet 10.61.4.194/26 brd 10.61.4.255 scope global qg-d6beba08-56
inet 10.61.4.195/32 brd 10.61.4.195 scope global qg-d6beba08-56
#
# Routing table (NETWORK NODE)
#
# ip netns exec qrouter-5078354f-0214-455d-bb4d-282b6d98fb86 ip r
default via 10.61.4.193 dev qg-d6beba08-56
10.61.4.192/26 dev qg-d6beba08-56 proto kernel scope link src 10.61.4.194
192.168.102.0/24 dev qr-aa72cd87-16 proto kernel scope link src
192.168.102.1
#
# Routing is working
#
# ip netns exec qrouter-5078354f-0214-455d-bb4d-282b6d98fb86 ip route get to
192.168.102.18
192.168.102.18 dev qr-aa72cd87-16 src 192.168.102.1
cache
# ip netns exec qrouter-5078354f-0214-455d-bb4d-282b6d98fb86 ip route get to
8.8.8.8
8.8.8.8 via 10.61.4.193 dev qg-d6beba08-56 src 10.61.4.194
cache
#
# Ping from qrouter namespace work both ways
#
# Goes through br-int -> br-tun -> . -> Compute-node -> VM
# ip netns exec qrouter-5078354f-0214-455d-bb4d-282b6d98fb86 ping
192.168.102.18
PING 192.168.102.18 (192.168.102.18) 56(84) bytes of data.
64 bytes from 192.168.102.18: icmp_seq=1 ttl=64 time=1.33 ms
# Goes through br-ex -> . -> 8.8.8.8
# ip netns exec qrouter-5078354f-0214-455d-bb4d-282b6d98fb86 ping 8.8.8.8
PING 8.8.8.8 (8.8.8.8) 56(84) bytes of data.
64 bytes from 8.8.8.8: icmp_seq=1 ttl=40 time=9.34 ms
#
# TCPDUMP (NETWORK NODE)
#
# ip netns exec qrouter-5078354f-0214-455d-bb4d-282b6d98fb86 tcpdump -eni
any icmp
tcpdump: verbose output suppressed, use -v or -vv for full protocol decode
listening on any, link-type LINUX_SLL (Linux cooked), capture size 65535
bytes
10:22:25.605747 In fa:16:3e:58:82:0c ethertype IPv4 (0x0800), length 100:
192.168.102.18 > 8.8.8.8: ICMP echo request, id 6972, seq 1548, length 64
10:22:25.605816 Out fa:16:3e:26:a7:52 ethertype IPv4 (0x0800), length 100:
10.61.4.195 > 8.8.8.8: ICMP echo request, id 6972, seq 1548, length 64
10:22:25.615094 M a7:00:81:00:10:2b ethertype IPv4 (0x0800), length 100:
8.8.8.8 > 10.61.4.195: ICMP echo reply, id 6972, seq 1548, length 64
#
# Bridges on network node
#
# !!!! enp21s0f0.43 is eth interface with configured vlan 43
#
ed943e95-fe6a-4760-8053-f178a9628087
Bridge br-int
fail_mode: secure
Port patch-tun
Interface patch-tun
type: patch
options: {peer=patch-int}
Port "tapd89063af-88"
tag: 1
Interface "tapd89063af-88"
type: internal
Port br-int
Interface br-int
type: internal
Port "qr-aa72cd87-16"
tag: 1
Interface "qr-aa72cd87-16"
type: internal
Port int-br-ex
Interface int-br-ex
Bridge br-ex
Port phy-br-ex
Interface phy-br-ex
Port "enp21s0f0.43"
Interface "enp21s0f0.43"
Port "qg-d6beba08-56"
Interface "qg-d6beba08-56"
type: internal
Port br-ex
Interface br-ex
type: internal
Bridge br-tun
Port "vxlan-c0a86402"
Interface "vxlan-c0a86402"
type: vxlan
options: {in_key=flow, local_ip="192.168.100.1",
out_key=flow, remote_ip="192.168.100.2"}
Port br-tun
Interface br-tun
type: internal
Port "vxlan-c0a86404"
Interface "vxlan-c0a86404"
type: vxlan
options: {in_key=flow, local_ip="192.168.100.1",
out_key=flow, remote_ip="192.168.100.4"}
Port patch-int
Interface patch-int
type: patch
options: {peer=patch-tun}
ovs_version: "2.1.3"
#
# BR-EX FLOWS (NETWORK NODE)
#
1(enp21s0f0.43): addr:00:10:18:9a:9d:94
3(qg-d6beba08-56): addr:00:10:18:9a:9d:94
6(phy-br-ex): addr:b6:1a:01:a3:3d:d4
LOCAL(br-ex): addr:00:10:18:9a:9d:94
NXST_FLOW reply (xid=0x4):
cookie=0x0, duration=6857.462s, table=0, n_packets=3141, n_bytes=285916,
idle_age=0, priority=1 actions=NORMAL
cookie=0x0, duration=6856.934s, table=0, n_packets=17, n_bytes=1330,
idle_age=1321, priority=2,in_port=6 actions=drop
#
# BR-INT FLOWS (NETWORK NODE)
#
3(tapd89063af-88): addr:0e:4e:e0:21:de:d9
4(qr-aa72cd87-16): addr:0e:4e:e0:21:de:d9
9(int-br-ex): addr:66:81:b9:c0:81:c0
10(patch-tun): addr:0e:4e:e0:21:de:d9
LOCAL(br-int): addr:3a:24:64:75:f6:40
NXST_FLOW reply (xid=0x4):
cookie=0x0, duration=6881.558s, table=0, n_packets=1459, n_bytes=138446,
idle_age=0, priority=1 actions=NORMAL
cookie=0x0, duration=6880.812s, table=0, n_packets=3197, n_bytes=291028,
idle_age=0, priority=2,in_port=9 actions=drop
cookie=0x0, duration=6881.511s, table=22, n_packets=0, n_bytes=0,
idle_age=6881, priority=0 actions=drop
#
# BR-TUN FLOWS (NETWORK NODE)
#
1(patch-int): addr:f2:b8:62:8a:cc:5f
2(vxlan-c0a86402): addr:5e:c6:a6:61:8b:67
3(vxlan-c0a86404): addr:5a:d8:91:57:f7:6c
LOCAL(br-tun): addr:da:a7:dc:d9:1c:41
NXST_FLOW reply (xid=0x4):
cookie=0x0, duration=7021.019s, table=0, n_packets=0, n_bytes=0,
idle_age=7021, priority=0 actions=drop
cookie=0x0, duration=7020.215s, table=0, n_packets=0, n_bytes=0,
idle_age=7020, priority=1,in_port=3 actions=resubmit(,3)
cookie=0x0, duration=7021.067s, table=0, n_packets=54, n_bytes=2716,
idle_age=1, priority=1,in_port=1 actions=resubmit(,1)
cookie=0x0, duration=7020.360s, table=0, n_packets=1554, n_bytes=149884,
idle_age=0, priority=1,in_port=2 actions=resubmit(,3)
cookie=0x0, duration=7020.918s, table=1, n_packets=3, n_bytes=126,
idle_age=3693, priority=1,dl_dst=01:00:00:00:00:00/01:00:00:00:00:00
actions=resubmit(,21)
cookie=0x0, duration=7020.968s, table=1, n_packets=51, n_bytes=2590,
idle_age=1, priority=1,dl_dst=00:00:00:00:00:00/01:00:00:00:00:00
actions=resubmit(,20)
cookie=0x0, duration=7020.868s, table=2, n_packets=0, n_bytes=0,
idle_age=7020, priority=0 actions=drop
cookie=0x0, duration=7020.819s, table=3, n_packets=0, n_bytes=0,
idle_age=7020, priority=0 actions=drop
cookie=0x0, duration=7019.502s, table=3, n_packets=1554, n_bytes=149884,
idle_age=0, priority=1,tun_id=0x10001 actions=mod_vlan_vid:1,resubmit(,10)
cookie=0x0, duration=7020.769s, table=10, n_packets=1554, n_bytes=149884,
idle_age=0, priority=1
actions=learn(table=20,hard_timeout=300,priority=1,NXM_OF_VLAN_TCI[0..11],NX
M_OF_ETH_DST[]=NXM_OF_ETH_SRC[],load:0->NXM_OF_VLAN_TCI[],load:NXM_NX_TUN_ID
[]->NXM_NX_TUN_ID[],output:NXM_OF_IN_PORT[]),output:1
cookie=0x0, duration=7020.720s, table=20, n_packets=0, n_bytes=0,
idle_age=7020, priority=0 actions=resubmit(,21)
cookie=0x0, duration=1491.217s, table=20, n_packets=0, n_bytes=0,
hard_timeout=300, idle_age=1491, hard_age=0,
priority=1,vlan_tci=0x0001/0x0fff,dl_dst=fa:16:3e:58:82:0c
actions=load:0->NXM_OF_VLAN_TCI[],load:0x10001->NXM_NX_TUN_ID[],output:2
cookie=0x0, duration=4926.186s, table=20, n_packets=45, n_bytes=2338,
idle_age=1, priority=2,dl_dst=fa:16:3e:58:82:0c
actions=load:0->NXM_OF_VLAN_TCI[],load:0x10001->NXM_NX_TUN_ID[],output:2
cookie=0x0, duration=7020.671s, table=21, n_packets=0, n_bytes=0,
idle_age=7020, priority=0 actions=drop
cookie=0x0, duration=7019.552s, table=21, n_packets=3, n_bytes=126,
idle_age=3693, dl_vlan=1
actions=strip_vlan,set_tunnel:0x10001,output:2,output:3
#
# Conntrack table
#
]# ip netns exec qrouter-5078354f-0214-455d-bb4d-282b6d98fb86 cat
/proc/net/nf_conntrack | grep 192.168.102.18
ipv4 2 icmp 1 29 src=192.168.102.18 dst=8.8.8.8 type=8 code=0
id=6972 src=8.8.8.8 dst=10.61.4.195 type=0 code=0 id=6972 mark=0 zone=0
use=2
#
# Kernel configuration
#
# FORWARDING
# sysctl -a | grep -E "ipv4.*forward" | grep -v "mc_"
net.ipv4.conf.all.forwarding = 1
net.ipv4.ip_forward = 1
net.ipv4.conf.default.forwarding = 1
net.ipv4.conf.br-ex.forwarding = 1
net.ipv4.conf.br-int.forwarding = 1
net.ipv4.conf.br-tun.forwarding = 1
net.ipv4.conf.enp21s0f0.forwarding = 1
net.ipv4.conf.enp21s0f0/40.forwarding = 1
net.ipv4.conf.enp21s0f0/41.forwarding = 1
net.ipv4.conf.enp21s0f0/42.forwarding = 1
net.ipv4.conf.enp21s0f0/43.forwarding = 1
net.ipv4.conf.enp21s0f1.forwarding = 1
net.ipv4.conf.int-br-enp21/43.forwarding = 1
net.ipv4.conf.int-br-ex.forwarding = 1
net.ipv4.conf.lo.forwarding = 1
net.ipv4.conf.ovs-system.forwarding = 1
net.ipv4.conf.phy-br-enp21/43.forwarding = 1
net.ipv4.conf.phy-br-ex.forwarding = 1
net.ipv4.conf.snooper0.forwarding = 1
net.ipv4.conf.snooper1.forwarding = 1
net.ipv4.conf.tapb4556004-a5.forwarding = 1
net.ipv4.conf.tapb4b327bf-e7.forwarding = 1
_______________________________________________
discuss mailing list
discuss@openvswitch.org
http://openvswitch.org/mailman/listinfo/discuss
