Adding the txt version of the pcaps
On Thu, 11 Apr 2024 at 19:30, Gavin McKee <gavmcke...@googlemail.com> wrote:
>
> Hi,
>
> I am a pretty bad issue where traffic is being NAT to the external_ip
> when it shouldn't be.
>
> First here are the versions of code I am running
> Version: (Open vSwitch) 3.2.2
> Version (OVN) 23.09.1
>
> I am using the Mellanox connect X6 and offloading everything to hardware.
>
> I am sending traffic between two VMs , they are connected to the same
> logical switch. The customer can ping between them on the private
> addresses - 172.27.34.210 -> 172.27.47.201 works fine
>
> compute_node_a (client)
> private_ip: 172.27.34.210
> public_ip: X.X.X.X
> ovn-vtep-if: 10.26.9.32/24
> external_mac: ae:29:e9:32:52:7f
> representor_port: enp148s0f1np1
> ofport: 623
> router_port_mac: 1a:b1:b6:11:6b:62
>
> compute_node_b (server)
> private_ip 172.27.47.201
> public_ip Y.Y.Y.Y
> ovn-vtep-if 10.26.17.28/24
> external_mac 1e:41:55:e6:8f:46
>
> However when the packet moves from the VM through the kernel path I
> can see the TCP SYN being sent on the representor see attached pcap
> file.
>
> Note the TCP source (52776) and destination ports (8080). You can see
> that the client sends a TCP SYN from 172.27.34.210 to 172.27.47.201 ,
> as it moves through the kernal (slow path) to the physical interface
> it NAT's to the dnat_and_snat external IP address. See the packet
> egress with the public IP address.
>
> I have no idea why this would happen on TCP only , ICMP is fine
>
> Output from an ofproto trace below
>
> ovs-appctl ofproto/trace br-int
> in_port=623,dl_src=ae:29:e9:32:52:7f,dl_dst=1e:41:55:e6:8f:46,tcp,nw_src=172.27.34.210,nw_dst=172.27.47.201,nw_ttl=32,tcp_src=52776,tcp_dst=8080,tcp_flags=2
> Flow:
> tcp,in_port=623,vlan_tci=0x0000,dl_src=ae:29:e9:32:52:7f,dl_dst=1e:41:55:e6:8f:46,nw_src=172.27.34.210,nw_dst=172.27.47.201,nw_tos=0,nw_ecn=0,nw_ttl=32,nw_frag=no,tp_src=52776,tp_dst=8080,tcp_flags=syn
>
> bridge("br-int")
> ----------------
> 0. in_port=623, priority 100, cookie 0x17928c
> set_field:0x78->reg13
> set_field:0xb5->reg11
> set_field:0x2ab->reg12
> set_field:0x1d7->metadata
> set_field:0x9->reg14
> resubmit(,8)
> 8. metadata=0x1d7, priority 50, cookie 0x78cd18a6
> set_field:0/0x1000->reg10
> resubmit(,73)
> 73.
> ip,reg14=0x9,metadata=0x1d7,dl_src=ae:29:e9:32:52:7f,nw_src=172.27.34.210,
> priority 90, cookie 0x17928c
> set_field:0/0x1000->reg10
> move:NXM_NX_REG10[12]->NXM_NX_XXREG0[111]
> -> NXM_NX_XXREG0[111] is now 0
> resubmit(,9)
> 9. metadata=0x1d7, priority 0, cookie 0x8830a146
> resubmit(,10)
> 10. metadata=0x1d7, priority 0, cookie 0x20463edc
> resubmit(,11)
> 11. metadata=0x1d7, priority 0, cookie 0x54ff7155
> resubmit(,12)
> 12. ip,metadata=0x1d7, priority 100, cookie 0xac030fc6
> set_field:0x1000000000000000000000000/0x1000000000000000000000000->xxreg0
> resubmit(,13)
> 13. metadata=0x1d7, priority 0, cookie 0x30de6504
> resubmit(,14)
> 14. ip,reg0=0x1/0x1,metadata=0x1d7, priority 100, cookie 0x3534a29
> ct(table=15,zone=NXM_NX_REG13[0..15])
> drop
> -> A clone of the packet is forked to recirculate. The forked
> pipeline will be resumed at table 15.
> -> Sets the packet to an untracked state, and clears all the
> conntrack fields.
>
> Final flow:
> tcp,reg0=0x1,reg11=0xb5,reg12=0x2ab,reg13=0x78,reg14=0x9,metadata=0x1d7,in_port=623,vlan_tci=0x0000,dl_src=ae:29:e9:32:52:7f,dl_dst=1e:41:55:e6:8f:46,nw_src=172.27.34.210,nw_dst=172.27.47.201,nw_tos=0,nw_ecn=0,nw_ttl=32,nw_frag=no,tp_src=52776,tp_dst=8080,tcp_flags=syn
> Megaflow:
> recirc_id=0,eth,tcp,in_port=623,dl_src=ae:29:e9:32:52:7f,dl_dst=1e:41:55:e6:8f:46,nw_src=172.27.34.210,nw_frag=no
> Datapath actions: ct(zone=120),recirc(0x26b18)
>
> ===============================================================================
> recirc(0x26b18) - resume conntrack with default ct_state=trk|new (use
> --ct-next to customize)
> ===============================================================================
>
> Flow:
> recirc_id=0x26b18,ct_state=new|trk,ct_zone=120,eth,tcp,reg0=0x1,reg11=0xb5,reg12=0x2ab,reg13=0x78,reg14=0x9,metadata=0x1d7,in_port=623,vlan_tci=0x0000,dl_src=ae:29:e9:32:52:7f,dl_dst=1e:41:55:e6:8f:46,nw_src=172.27.34.210,nw_dst=172.27.47.201,nw_tos=0,nw_ecn=0,nw_ttl=32,nw_frag=no,tp_src=52776,tp_dst=8080,tcp_flags=syn
>
> bridge("br-int")
> ----------------
> thaw
> Resuming from table 15
> 15. ct_state=+new-est+trk,metadata=0x1d7, priority 7, cookie 0x6e0867dd
>
> set_field:0x80000000000000000000000000/0x80000000000000000000000000->xxreg0
>
> set_field:0x200000000000000000000000000/0x200000000000000000000000000->xxreg0
> resubmit(,16)
> 16. conj_id=3485242595,tcp,reg0=0x80/0x80,reg14=0x9,metadata=0x1d7,
> priority 3000, cookie 0x52a834a1
> set_field:0x1000000000000/0x1000000000000->xreg4
> set_field:0x2000000000000000000000000/0x2000000000000000000000000->xxreg0
> resubmit(,17)
> 17. reg8=0x10000/0x10000,metadata=0x1d7, priority 1000, cookie 0xa729a095
> set_field:0/0x1000000000000->xreg4
> set_field:0/0x2000000000000->xreg4
> set_field:0/0x4000000000000->xreg4
> resubmit(,18)
> 18. metadata=0x1d7, priority 0, cookie 0x96baa2f
> resubmit(,19)
> 19. metadata=0x1d7, priority 0, cookie 0x3216af2f
> resubmit(,20)
> 20. metadata=0x1d7, priority 0, cookie 0x9e83bb36
> resubmit(,21)
> 21. metadata=0x1d7, priority 0, cookie 0x3fc32b6a
> resubmit(,22)
> 22. metadata=0x1d7, priority 0, cookie 0x841766eb
> resubmit(,23)
> 23. metadata=0x1d7, priority 0, cookie 0x6ea80e05
> resubmit(,24)
> 24. metadata=0x1d7, priority 0, cookie 0x4d31ab9
> resubmit(,25)
> 25. metadata=0x1d7, priority 0, cookie 0xfe78236
> resubmit(,26)
> 26. metadata=0x1d7, priority 0, cookie 0x2b009c14
> resubmit(,27)
> 27. metadata=0x1d7, priority 0, cookie 0x7b6bf676
> set_field:0/0x1000000000000->xreg4
> set_field:0/0x2000000000000->xreg4
> set_field:0/0x4000000000000->xreg4
> resubmit(,28)
> 28. ip,reg0=0x2/0x2002,metadata=0x1d7, priority 100, cookie 0x36a19530
>
> ct(commit,zone=NXM_NX_REG13[0..15],nat(src),exec(set_field:0/0x1->ct_mark))
> nat(src)
> set_field:0/0x1->ct_mark
> -> Sets the packet to an untracked state, and clears all the
> conntrack fields.
> resubmit(,29)
> 29. metadata=0x1d7, priority 0, cookie 0xa36067fa
> resubmit(,30)
> 30. metadata=0x1d7, priority 0, cookie 0xb56c19e
> resubmit(,31)
> 31. metadata=0x1d7, priority 0, cookie 0x2d9fecef
> resubmit(,32)
> 32. metadata=0x1d7, priority 0, cookie 0xad0aad2f
> resubmit(,33)
> 33. metadata=0x1d7, priority 0, cookie 0x4e92d423
> resubmit(,34)
> 34. metadata=0x1d7, priority 0, cookie 0x9fe31110
> resubmit(,35)
> 35. metadata=0x1d7,dl_dst=1e:41:55:e6:8f:46, priority 50, cookie 0x6ea924d2
> set_field:0x15->reg15
> resubmit(,37)
> 37. priority 0
> resubmit(,39)
> 39. reg15=0x15,metadata=0x1d7, priority 100, cookie 0xcdd83576
> set_field:0x1d7/0xffffff->tun_id
> set_field:0x15->tun_metadata0
> move:NXM_NX_REG14[0..14]->NXM_NX_TUN_METADATA0[16..30]
> -> NXM_NX_TUN_METADATA0[16..30] is now 0x9
> output:777
> -> output to kernel tunnel
> resubmit(,40)
> 40. priority 0
> drop
>
> Final flow:
> recirc_id=0x26b18,eth,tcp,reg0=0x283,reg11=0xb5,reg12=0x2ab,reg13=0x78,reg14=0x9,reg15=0x15,tun_id=0x1d7,metadata=0x1d7,in_port=623,vlan_tci=0x0000,dl_src=ae:29:e9:32:52:7f,dl_dst=1e:41:55:e6:8f:46,nw_src=172.27.34.210,nw_dst=172.27.47.201,nw_tos=0,nw_ecn=0,nw_ttl=32,nw_frag=no,tp_src=52776,tp_dst=8080,tcp_flags=syn
> Megaflow:
> recirc_id=0x26b18,ct_state=+new-est-rel-rpl-inv+trk,ct_mark=0/0x1,eth,tcp,in_port=623,dl_src=ae:29:e9:32:52:7f,dl_dst=1e:41:55:e6:8f:46,nw_src=172.27.32.0/19,nw_dst=172.27.40.0/21,nw_ecn=0,nw_frag=no,tp_src=0x8000/0x8000,tp_dst=0x1000/0xf000
> Datapath actions:
> ct(commit,zone=120,mark=0/0x1,nat(src)),set(tunnel(tun_id=0x1d7,dst=10.26.17.28,ttl=64,tp_dst=6081,geneve({class=0x102,type=0x80,len=4,0x90015}),flags(df|csum|key))),2
No. Time Source Destination Protocol
Length Info Delta
1 16:51:46.920839 216.86.161.7 172.27.26.218 TCP
70 52776 → 8080 [SYN] Seq=0 Win=42340 Len=0 MSS=1460 SACK_PERM WS=4096
0.000000
Frame 1: 70 bytes on wire (560 bits), 70 bytes captured (560 bits)
Ethernet II, Src: ae:29:e9:32:52:7f (ae:29:e9:32:52:7f), Dst: IETF-VRRP-VRID_04
(00:00:5e:00:01:04)
802.1Q Virtual LAN, PRI: 0, DEI: 0, ID: 120
Internet Protocol Version 4, Src: 216.86.161.7, Dst: 172.27.26.218
Transmission Control Protocol, Src Port: 52776, Dst Port: 8080, Seq: 0, Len: 0
No. Time Source Destination Protocol
Length Info Delta
2 16:51:55.112855 216.86.161.7 172.27.26.218 TCP
70 [TCP Retransmission] 52776 → 8080 [SYN] Seq=0 Win=42340 Len=0 MSS=1460
SACK_PERM WS=4096 8.192016
Frame 2: 70 bytes on wire (560 bits), 70 bytes captured (560 bits)
Ethernet II, Src: ae:29:e9:32:52:7f (ae:29:e9:32:52:7f), Dst: IETF-VRRP-VRID_04
(00:00:5e:00:01:04)
802.1Q Virtual LAN, PRI: 0, DEI: 0, ID: 120
Internet Protocol Version 4, Src: 216.86.161.7, Dst: 172.27.26.218
Transmission Control Protocol, Src Port: 52776, Dst Port: 8080, Seq: 0, Len: 0
No. Time Source Destination Protocol
Length Info Delta
3 16:52:11.245156 216.86.161.7 172.27.26.218 TCP
70 [TCP Retransmission] 52776 → 8080 [SYN] Seq=0 Win=42340 Len=0 MSS=1460
SACK_PERM WS=4096 16.132301
Frame 3: 70 bytes on wire (560 bits), 70 bytes captured (560 bits)
Ethernet II, Src: ae:29:e9:32:52:7f (ae:29:e9:32:52:7f), Dst: IETF-VRRP-VRID_04
(00:00:5e:00:01:04)
802.1Q Virtual LAN, PRI: 0, DEI: 0, ID: 120
Internet Protocol Version 4, Src: 216.86.161.7, Dst: 172.27.26.218
Transmission Control Protocol, Src Port: 52776, Dst Port: 8080, Seq: 0, Len: 0
No. Time Source Destination Protocol
Length Info Delta
4 16:52:44.270586 216.86.161.7 172.27.26.218 TCP
70 [TCP Retransmission] 52776 → 8080 [SYN] Seq=0 Win=42340 Len=0 MSS=1460
SACK_PERM WS=4096 33.025430
Frame 4: 70 bytes on wire (560 bits), 70 bytes captured (560 bits)
Ethernet II, Src: ae:29:e9:32:52:7f (ae:29:e9:32:52:7f), Dst: IETF-VRRP-VRID_04
(00:00:5e:00:01:04)
802.1Q Virtual LAN, PRI: 0, DEI: 0, ID: 120
Internet Protocol Version 4, Src: 216.86.161.7, Dst: 172.27.26.218
Transmission Control Protocol, Src Port: 52776, Dst Port: 8080, Seq: 0, Len: 0
No. Time Source Destination Protocol
Length Info Delta
5 16:53:54.812795 216.86.161.7 172.27.26.218 TCP
70 49878 → 8080 [SYN] Seq=0 Win=42340 Len=0 MSS=1460 SACK_PERM WS=4096
70.542209
Frame 5: 70 bytes on wire (560 bits), 70 bytes captured (560 bits)
Ethernet II, Src: ae:29:e9:32:52:7f (ae:29:e9:32:52:7f), Dst: IETF-VRRP-VRID_04
(00:00:5e:00:01:04)
802.1Q Virtual LAN, PRI: 0, DEI: 0, ID: 120
Internet Protocol Version 4, Src: 216.86.161.7, Dst: 172.27.26.218
Transmission Control Protocol, Src Port: 49878, Dst Port: 8080, Seq: 0, Len: 0
No. Time Source Destination Protocol
Length Info Delta
6 16:53:55.817151 216.86.161.7 172.27.26.218 TCP
70 [TCP Retransmission] 49878 → 8080 [SYN] Seq=0 Win=42340 Len=0 MSS=1460
SACK_PERM WS=4096 1.004356
Frame 6: 70 bytes on wire (560 bits), 70 bytes captured (560 bits)
Ethernet II, Src: ae:29:e9:32:52:7f (ae:29:e9:32:52:7f), Dst: IETF-VRRP-VRID_04
(00:00:5e:00:01:04)
802.1Q Virtual LAN, PRI: 0, DEI: 0, ID: 120
Internet Protocol Version 4, Src: 216.86.161.7, Dst: 172.27.26.218
Transmission Control Protocol, Src Port: 49878, Dst Port: 8080, Seq: 0, Len: 0
No. Time Source Destination Protocol
Length Info Delta
7 16:53:57.833141 216.86.161.7 172.27.26.218 TCP
70 [TCP Retransmission] 49878 → 8080 [SYN] Seq=0 Win=42340 Len=0 MSS=1460
SACK_PERM WS=4096 2.015990
Frame 7: 70 bytes on wire (560 bits), 70 bytes captured (560 bits)
Ethernet II, Src: ae:29:e9:32:52:7f (ae:29:e9:32:52:7f), Dst: IETF-VRRP-VRID_04
(00:00:5e:00:01:04)
802.1Q Virtual LAN, PRI: 0, DEI: 0, ID: 120
Internet Protocol Version 4, Src: 216.86.161.7, Dst: 172.27.26.218
Transmission Control Protocol, Src Port: 49878, Dst Port: 8080, Seq: 0, Len: 0
No. Time Source Destination Protocol
Length Info Delta
8 16:54:02.089165 216.86.161.7 172.27.26.218 TCP
70 [TCP Retransmission] 49878 → 8080 [SYN] Seq=0 Win=42340 Len=0 MSS=1460
SACK_PERM WS=4096 4.256024
Frame 8: 70 bytes on wire (560 bits), 70 bytes captured (560 bits)
Ethernet II, Src: ae:29:e9:32:52:7f (ae:29:e9:32:52:7f), Dst: IETF-VRRP-VRID_04
(00:00:5e:00:01:04)
802.1Q Virtual LAN, PRI: 0, DEI: 0, ID: 120
Internet Protocol Version 4, Src: 216.86.161.7, Dst: 172.27.26.218
Transmission Control Protocol, Src Port: 49878, Dst Port: 8080, Seq: 0, Len: 0
No. Time Source Destination Protocol
Length Info Delta
1 16:51:46.920820 172.27.34.210 172.27.26.218 TCP
66 52776 → 8080 [SYN] Seq=0 Win=42340 Len=0 MSS=1460 SACK_PERM WS=4096
0.000000
Frame 1: 66 bytes on wire (528 bits), 66 bytes captured (528 bits)
Ethernet II, Src: ae:29:e9:32:52:7f (ae:29:e9:32:52:7f), Dst: 1a:b1:b6:11:6b:62
(1a:b1:b6:11:6b:62)
Internet Protocol Version 4, Src: 172.27.34.210, Dst: 172.27.26.218
Transmission Control Protocol, Src Port: 52776, Dst Port: 8080, Seq: 0, Len: 0
No. Time Source Destination Protocol
Length Info Delta
2 16:51:55.112837 172.27.34.210 172.27.26.218 TCP
66 [TCP Retransmission] 52776 → 8080 [SYN] Seq=0 Win=42340 Len=0 MSS=1460
SACK_PERM WS=4096 8.192017
Frame 2: 66 bytes on wire (528 bits), 66 bytes captured (528 bits)
Ethernet II, Src: ae:29:e9:32:52:7f (ae:29:e9:32:52:7f), Dst: 1a:b1:b6:11:6b:62
(1a:b1:b6:11:6b:62)
Internet Protocol Version 4, Src: 172.27.34.210, Dst: 172.27.26.218
Transmission Control Protocol, Src Port: 52776, Dst Port: 8080, Seq: 0, Len: 0
No. Time Source Destination Protocol
Length Info Delta
3 16:52:11.240884 172.27.34.210 172.27.26.218 TCP
66 [TCP Retransmission] 52776 → 8080 [SYN] Seq=0 Win=42340 Len=0 MSS=1460
SACK_PERM WS=4096 16.128047
Frame 3: 66 bytes on wire (528 bits), 66 bytes captured (528 bits)
Ethernet II, Src: ae:29:e9:32:52:7f (ae:29:e9:32:52:7f), Dst: 1a:b1:b6:11:6b:62
(1a:b1:b6:11:6b:62)
Internet Protocol Version 4, Src: 172.27.34.210, Dst: 172.27.26.218
Transmission Control Protocol, Src Port: 52776, Dst Port: 8080, Seq: 0, Len: 0
No. Time Source Destination Protocol
Length Info Delta
4 16:52:44.264972 172.27.34.210 172.27.26.218 TCP
66 [TCP Retransmission] 52776 → 8080 [SYN] Seq=0 Win=42340 Len=0 MSS=1460
SACK_PERM WS=4096 33.024088
Frame 4: 66 bytes on wire (528 bits), 66 bytes captured (528 bits)
Ethernet II, Src: ae:29:e9:32:52:7f (ae:29:e9:32:52:7f), Dst: 1a:b1:b6:11:6b:62
(1a:b1:b6:11:6b:62)
Internet Protocol Version 4, Src: 172.27.34.210, Dst: 172.27.26.218
Transmission Control Protocol, Src Port: 52776, Dst Port: 8080, Seq: 0, Len: 0
No. Time Source Destination Protocol
Length Info Delta
5 16:53:54.808694 172.27.34.210 172.27.26.218 TCP
66 49878 → 8080 [SYN] Seq=0 Win=42340 Len=0 MSS=1460 SACK_PERM WS=4096
70.543722
Frame 5: 66 bytes on wire (528 bits), 66 bytes captured (528 bits)
Ethernet II, Src: ae:29:e9:32:52:7f (ae:29:e9:32:52:7f), Dst: 1a:b1:b6:11:6b:62
(1a:b1:b6:11:6b:62)
Internet Protocol Version 4, Src: 172.27.34.210, Dst: 172.27.26.218
Transmission Control Protocol, Src Port: 49878, Dst Port: 8080, Seq: 0, Len: 0
No. Time Source Destination Protocol
Length Info Delta
6 16:53:55.817135 172.27.34.210 172.27.26.218 TCP
66 [TCP Retransmission] 49878 → 8080 [SYN] Seq=0 Win=42340 Len=0 MSS=1460
SACK_PERM WS=4096 1.008441
Frame 6: 66 bytes on wire (528 bits), 66 bytes captured (528 bits)
Ethernet II, Src: ae:29:e9:32:52:7f (ae:29:e9:32:52:7f), Dst: 1a:b1:b6:11:6b:62
(1a:b1:b6:11:6b:62)
Internet Protocol Version 4, Src: 172.27.34.210, Dst: 172.27.26.218
Transmission Control Protocol, Src Port: 49878, Dst Port: 8080, Seq: 0, Len: 0
No. Time Source Destination Protocol
Length Info Delta
7 16:53:57.833131 172.27.34.210 172.27.26.218 TCP
66 [TCP Retransmission] 49878 → 8080 [SYN] Seq=0 Win=42340 Len=0 MSS=1460
SACK_PERM WS=4096 2.015996
Frame 7: 66 bytes on wire (528 bits), 66 bytes captured (528 bits)
Ethernet II, Src: ae:29:e9:32:52:7f (ae:29:e9:32:52:7f), Dst: 1a:b1:b6:11:6b:62
(1a:b1:b6:11:6b:62)
Internet Protocol Version 4, Src: 172.27.34.210, Dst: 172.27.26.218
Transmission Control Protocol, Src Port: 49878, Dst Port: 8080, Seq: 0, Len: 0
No. Time Source Destination Protocol
Length Info Delta
8 16:54:02.089151 172.27.34.210 172.27.26.218 TCP
66 [TCP Retransmission] 49878 → 8080 [SYN] Seq=0 Win=42340 Len=0 MSS=1460
SACK_PERM WS=4096 4.256020
Frame 8: 66 bytes on wire (528 bits), 66 bytes captured (528 bits)
Ethernet II, Src: ae:29:e9:32:52:7f (ae:29:e9:32:52:7f), Dst: 1a:b1:b6:11:6b:62
(1a:b1:b6:11:6b:62)
Internet Protocol Version 4, Src: 172.27.34.210, Dst: 172.27.26.218
Transmission Control Protocol, Src Port: 49878, Dst Port: 8080, Seq: 0, Len: 0
_______________________________________________
discuss mailing list
disc...@openvswitch.org
https://mail.openvswitch.org/mailman/listinfo/ovs-discuss
