Hi,
I'm trying to debug why a flow is not offloaded when the traffic is routing
through an external GW.
I'm using the Mellanox ConnectX-6 Card, here is some information relating
to my setup.
Hypervisor usc01a-032-16a ovn-encap-ip="192.168.1.1"
Virtual Machine MAC: 10:70:fd:8a:a1:00
Virtual Machine IP : 10.0.0.11/24
DNAT / SNAT: 216.147.118.11
Destination External GW usc01a-032-19b ovn-encap-ip="192.168.0.20"
Destination IP: 10.10.10.10
You can see here that I can ping the server on the external network
gmckee@usc01a-032-16a-vm1:~$ ping 10.10.10.10
PING 10.10.10.10 (10.10.10.10) 56(84) bytes of data.
64 bytes from 10.10.10.10: icmp_seq=1 ttl=62 time=96.5 ms
64 bytes from 10.10.10.10: icmp_seq=2 ttl=62 time=1.12 ms
64 bytes from 10.10.10.10: icmp_seq=3 ttl=62 time=0.563 ms
64 bytes from 10.10.10.10: icmp_seq=4 ttl=62 time=0.551 ms
64 bytes from 10.10.10.10: icmp_seq=5 ttl=62 time=0.558 ms
64 bytes from 10.10.10.10: icmp_seq=6 ttl=62 time=0.504 ms
TCP Dump on the server showing the NAT address and the response to the ICMP
root@usc01a-032-19d:/home/gmckee# tcpdump -ni enp129s0f1 icmp
tcpdump: verbose output suppressed, use -v[v]... for full protocol decode
listening on enp129s0f1, link-type EN10MB (Ethernet), snapshot length
262144 bytes
23:27:59.934875 IP 216.147.118.11 > 10.10.10.10: ICMP echo request, id 67,
seq 1, length 64
23:27:59.934935 IP 10.10.10.10 > 216.147.118.11: ICMP echo reply, id 67,
seq 1, length 64
23:28:00.907343 IP 216.147.118.11 > 10.10.10.10: ICMP echo request, id 67,
seq 2, length 64
23:28:00.907368 IP 10.10.10.10 > 216.147.118.11: ICMP echo reply, id 67,
seq 2, length 64
23:28:01.908366 IP 216.147.118.11 > 10.10.10.10: ICMP echo request, id 67,
seq 3, length 64
23:28:01.908392 IP 10.10.10.10 > 216.147.118.11: ICMP echo reply, id 67,
seq 3, length 64
23:28:02.916122 IP 216.147.118.11 > 10.10.10.10: ICMP echo request, id 67,
seq 4, length 64
23:28:02.916147 IP 10.10.10.10 > 216.147.118.11: ICMP echo reply, id 67,
seq 4, length 64
23:28:03.940096 IP 216.147.118.11 > 10.10.10.10: ICMP echo request, id 67,
seq 5, length 64
23:28:03.940123 IP 10.10.10.10 > 216.147.118.11: ICMP echo reply, id 67,
seq 5, length 64
Looking at the output from the ovs data plane
root@usc01a-032-19b:/home/gmckee# ovs-appctl dpctl/dump-flows -m
type=non-offloaded
ufid:a8bfb809-4ab7-4500-b742-113d579ad43c,
skb_priority(0/0),skb_mark(0/0),ct_state(0/0),ct_zone(0/0),ct_mark(0/0),ct_label(0/0),recirc_id(0x48),dp_hash(0/0),in_port(enp1s0f0),packet_type(ns=0/0,id=0/0),eth(src=74:83:ef:d8:18:37,dst=00:00:20:20:12:13),eth_type(0x0800),ipv4(src=
0.0.0.0/0.0.0.0,dst=10.0.0.11,proto=1,tos=0/0x3,ttl=63,frag=no),icmp(type=0/0,code=0/0),
packets:70, bytes:5880, used:0.970s, dp:tc,
actions:ct_clear,set(tunnel(tun_id=0x1,dst=192.168.1.1,ttl=64,tp_dst=6081,geneve({class=0x102,type=0x80,len=4,0xa0001}),flags(csum|key))),set(eth(src=00:00:00:00:00:01,dst=10:70:fd:8a:a1:00)),set(ipv4(ttl=62)),genev_sys_6081
ufid:7a9f9e55-5b4d-4774-bea9-2ffe7f38e2d9,
skb_priority(0/0),tunnel(tun_id=0x3,src=192.168.1.1,dst=192.168.0.20,ttl=0/0,tp_dst=6081,geneve({class=0x102/0,type=0x80/0,len=4,0x10004/0}),flags(+key)),skb_mark(0/0),ct_state(0/0),ct_zone(0/0),ct_mark(0/0),ct_label(0/0),recirc_id(0x46),dp_hash(0/0),in_port(genev_sys_6081),packet_type(ns=0/0,id=0/0),eth(src=00:00:20:20:12:13,dst=74:83:ef:d8:18:37),eth_type(0x0800),ipv4(src=
128.0.0.0/128.0.0.0,dst=0.0.0.0/128.0.0.0,proto=0/0,tos=0/0,ttl=0/0,frag=no),
packets:69, bytes:5796, used:0.970s, dp:tc, actions:ct_clear,enp1s0f0
ufid:56cc48cc-370f-4406-81ac-08637e9f53d5,
recirc_id(0),dp_hash(0/0),skb_priority(0/0),in_port(enp1s0f0),skb_mark(0/0),ct_state(0/0),ct_zone(0/0),ct_mark(0/0),ct_label(0/0),eth(src=74:83:ef:d8:18:68,dst=01:80:c2:00:00:00),eth_type(0/0xffff),
packets:12362, bytes:1471078, used:0.601s, dp:ovs, actions:drop
root@usc01a-032-19b:/home/gmckee# ovs-appctl dpctl/dump-flows -m
type=offloaded
ufid:1dd673ce-f923-4792-b872-440bd7c3899c,
skb_priority(0/0),skb_mark(0/0),ct_state(0/0),ct_zone(0/0),ct_mark(0/0),ct_label(0/0),recirc_id(0),dp_hash(0/0),in_port(enp1s0f0),packet_type(ns=0/0,id=0/0),eth(src=74:83:ef:d8:18:68,dst=01:80:c2:00:00:0e),eth_type(0x88cc),
packets:0, bytes:0, used:1.140s, offloaded:yes, dp:tc, actions:drop
ufid:a76f46b4-73a3-46f3-9fc3-0c05d8bc102c,
skb_priority(0/0),skb_mark(0/0),ct_state(0/0),ct_zone(0/0),ct_mark(0/0),ct_label(0/0),recirc_id(0),dp_hash(0/0),in_port(enp1s0f0),packet_type(ns=0/0,id=0/0),eth(src=74:83:ef:d8:18:37,dst=00:00:20:20:12:13),eth_type(0x0800),ipv4(src=
10.8.0.0/255.248.0.0,dst=216.147.118.11,proto=1,tos=0/0,ttl=63,frag=no),icmp(type=0/0,code=0/0),
packets:78, bytes:6552, used:0.200s, offloaded:yes, dp:tc,
actions:ct(zone=4,nat),recirc(0x48)
ufid:41aaf6f4-c74d-45e5-a5bc-8b8beb742923,
skb_priority(0/0),tunnel(tun_id=0x3,src=192.168.1.1,dst=192.168.0.20,ttl=0/0,tp_dst=6081,geneve({class=0x102,type=0x80,len=4,0x10004/0x7fffffff}),flags(+key)),skb_mark(0/0),ct_state(0/0),ct_zone(0/0),ct_mark(0/0),ct_label(0/0),recirc_id(0),dp_hash(0/0),in_port(genev_sys_6081),packet_type(ns=0/0,id=0/0),eth(src=00:00:00:00:00:00/00:00:00:00:00:00,dst=00:00:00:00:00:00/00:00:00:00:00:00),eth_type(0x0800),ipv4(src=10.0.0.11,dst=
0.0.0.0/128.0.0.0,proto=0/0,tos=0/0,ttl=0/0,frag=no), packets:78,
bytes:6552, used:0.200s, offloaded:yes, dp:tc,
actions:ct(zone=4,nat),recirc(0x46)
ufid:4f3eb036-b99b-4f32-87c3-7f32d1a4ebe7,
skb_priority(0/0),skb_mark(0/0),ct_state(0/0),ct_zone(0/0),ct_mark(0/0),ct_label(0/0),recirc_id(0),dp_hash(0/0),in_port(public),packet_type(ns=0/0,id=0/0),eth(src=10:70:fd:df:a5:6e,dst=01:80:c2:00:00:0e),eth_type(0x88cc),
packets:0, bytes:0, used:7.990s, offloaded:yes, dp:tc, actions:drop
Looking at the output from tc monitor
replaced filter dev genev_sys_6081 ingress protocol ip pref 3 flower chain
70 handle 0x1
eth_type ipv4
src_ip 10.0.0.11
enc_dst_ip 192.168.0.20
enc_src_ip 192.168.1.1
enc_key_id 3
enc_dst_port 6081
enc_tos 0
geneve_opts 0102:80:00010004/0000:00:00000000
ip_flags nofrag
in_hw in_hw_count 1
action order 1: ct commit zone 4 nat src addr 216.147.118.11 pipe
index 2 ref 1 bind 1
cookie c219bd3d4b45df0aa852f681ef161279
used_hw_stats delayed
action order 2: gact action goto chain 73
random type none pass val 0
index 4 ref 1 bind 1
cookie c219bd3d4b45df0aa852f681ef161279
used_hw_stats delayed
added chain dev genev_sys_6081 parent ffff: chain 73
replaced filter dev genev_sys_6081 ingress protocol ip pref 3 flower chain
73 handle 0x1
dst_mac 74:83:ef:d8:18:37
src_mac 00:00:20:20:12:13
eth_type ipv4
dst_ip 10.10.10.10/1
enc_dst_ip 192.168.0.20
enc_src_ip 192.168.1.1
enc_key_id 3
enc_dst_port 6081
enc_tos 0
geneve_opts 0102:80:00010004/0000:00:00000000
ip_flags nofrag
not_in_hw
action order 1: ct clear pipe
index 3 ref 1 bind 1
cookie db0fe088674be108cac024895b5b8856
action order 2: tunnel_key unset pipe
index 1 ref 1 bind 1
no_percpu
action order 3: mirred (Egress Redirect to device enp1s0f0) stolen
index 1 ref 1 bind 1
cookie db0fe088674be108cac024895b5b8856
no_percpu
When I do a TCP dump on the genev_sys_6081, I see all the traffic hitting
the CPU in the kernal .
root@usc01a-032-19b:/home/gmckee# tcpdump -i genev_sys_6081
tcpdump: verbose output suppressed, use -v[v]... for full protocol decode
listening on genev_sys_6081, link-type EN10MB (Ethernet), snapshot length
262144 bytes
23:31:54.177363 IP 10.0.0.11 > 10.10.10.10: ICMP echo request, id 68, seq
99, length 64
23:31:54.177676 IP 10.10.10.10 > 10.0.0.11: ICMP echo reply, id 68, seq 99,
length 64
23:31:55.201253 IP 10.0.0.11 > 10.10.10.10: ICMP echo request, id 68, seq
100, length 64
23:31:55.201566 IP 10.10.10.10 > 10.0.0.11: ICMP echo reply, id 68, seq
100, length 64
23:31:56.225344 IP 10.0.0.11 > 10.10.10.10: ICMP echo request, id 68, seq
101, length 64
23:31:56.225654 IP 10.10.10.10 > 10.0.0.11: ICMP echo reply, id 68, seq
101, length 64
23:31:57.249252 IP 10.0.0.11 > 10.10.10.10: ICMP echo request, id 68, seq
102, length 64
23:31:57.249498 IP 10.10.10.10 > 10.0.0.11: ICMP echo reply, id 68, seq
102, length 64
_______________________________________________
discuss mailing list
disc...@openvswitch.org
https://mail.openvswitch.org/mailman/listinfo/ovs-discuss
