On 11/8/24 11:15, Krzysztof Tomaszewski via discuss wrote:
> Hello,
>
> I am trying to trace ICMP ping reply packet on DPDK + bond gateway node and I
> am struggling.
> It looks like ovs-appctl ofproto/trace is not applying conntrack base data
> and in trace flow
> packet is not unSNAT-ed.
>
> Of course, ICMP ping is working properly - packet is going back to the VM -
> the problem seems to be with
> ofproto/trace only.
>
> Infrastructure: OpenStack VM 10.44.1.174 with tenant network only (no FIP)
> that pings 8.8.8.8
> and router with external gateway enabled with 194.146.52.200 gw IP that is on
> network002 node.
> Network Node/gw chassis: OVN v24.03.2, OVS v3.3.2, DPDK 23.11.0.
>
> [root@network002 /]# flow=$(ovs-tcpdump -nXX -i dpdkbond -c 1 vlan 170 and
> host 194.146.52.200 and icmp and 'icmp[0]=0' | ovs-tcpundump)
> [root@network002 /]# echo $flow
> fa163e80daf6b0cf0e043dfd810000aa0800450000540000000075013e3f08080808c29234c8000054450e8288cc38d82d6700000000e3590c0000000000101112131415161718191a1b1c1d1e1f202122232425262728292a2b2c2d2e2f3031323334353637
>
> ####
> [root@network002 /]# ovs-appctl ofproto/trace br-ex in_port=2 $flow
> Flow:
> icmp,in_port=2,dl_vlan=170,dl_vlan_pcp=0,vlan_tci1=0x0000,dl_src=b0:cf:0e:04:3d:fd,dl_dst=fa:16:3e:80:da:f6,nw_src=8.8.8.8,nw_dst=194.146.52.200,nw_tos=0,nw_ecn=0,nw_ttl=117,nw_frag=no,icmp_type=0,icmp_code=0
>
> <CUT>
>
> 12. ip,reg14=0x1,metadata=0xa5,nw_dst=194.146.52.200, priority 100, cookie
> 0xb8e323f4
> ct(table=13,zone=NXM_NX_REG12[0..15],nat)
> nat
> -> A clone of the packet is forked to recirculate. The forked pipeline
> will be resumed at table 13.
> -> Sets the packet to an untracked state, and clears all the conntrack
> fields.
>
> Final flow: unchanged
> Megaflow:
> recirc_id=0,eth,icmp,in_port=2,dl_vlan=170,dl_vlan_pcp=0,dl_src=b0:cf:0e:04:3d:fd,dl_dst=fa:16:3e:80:da:f6,nw_src=8.0.0.0/7,nw_dst=194.146.52.200,nw_ttl=117,nw_frag=no,icmp_type=0x0/0xf8
> Datapath actions: pop_vlan,ct(zone=171,nat),recirc(0x130b9b)
>
> ===============================================================================
> recirc(0x130b9b) - resume conntrack with default ct_state=trk|new (use
> --ct-next to customize)
> Replacing src/dst IP/ports to simulate NAT:
> Initial flow:
> Modified flow:
> ===============================================================================
>
> Flow:
> recirc_id=0x130b9b,ct_state=new|trk,ct_zone=171,eth,icmp,reg0=0xfa16,reg1=0x3e80daf6,reg9=0x4,reg11=0x97,reg12=0xab,reg14=0x1,metadata=0xa5,in_port=3,vlan_tci=0x0000,dl_src=b0:cf:0e:04:3d:fd,dl_dst=fa:16:3e:80:da:f6,nw_src=8.8.8.8,nw_dst=194.146.52.200,nw_tos=0,nw_ecn=0,nw_ttl=117,nw_frag=no,icmp_type=0,icmp_code=0
>
>
> <CUT>
>
> 20. metadata=0xa5, priority 0, cookie 0xcbf3e7a5
> set_field:0/0xffffffff->xxreg1
> resubmit(,21)
> 21. ip,metadata=0xa5,nw_dst=194.146.52.128/25, priority 77, cookie 0xcdd1182
> dec_ttl()
> set_field:0/0xffff00000000->xreg4
> move:NXM_OF_IP_DST[]->NXM_NX_XXREG0[96..127]
> -> NXM_NX_XXREG0[96..127] is now 0xc29234c8
> set_field:0xc29234c80000000000000000/0xffffffff0000000000000000->xxreg0
> set_field:fa:16:3e:80:da:f6->eth_src
> set_field:0x1->reg15
> set_field:0x1/0x1->reg10
> resubmit(,22)
> 22. reg8=0/0xffff,metadata=0xa5, priority 150, cookie 0x2d387d2e
> resubmit(,23)
> 23. metadata=0xa5, priority 0, cookie 0x87eb379e
> set_field:0/0xffff00000000->xreg4
> resubmit(,24)
> 24. reg8=0/0xffff,metadata=0xa5, priority 150, cookie 0x129f6153
> resubmit(,25)
> 25. ip,reg14=0x1,reg15=0x1,metadata=0xa5,nw_dst=194.146.52.200, priority 150,
> cookie 0xd283cb8a
> drop
>
> Final flow:
> recirc_id=0x130b9b,ct_state=new|trk,ct_zone=171,eth,icmp,reg0=0xc29234c8,reg1=0xc29234c8,reg9=0x4,reg10=0x1,reg11=0x97,reg12=0xab,reg14=0x1,reg15=0x1,metadata=0xa5,in_port=3,vlan_tci=0x0000,dl_src=fa:16:3e:80:da:f6,dl_dst=fa:16:3e:80:da:f6,nw_src=8.8.8.8,nw_dst=194.146.52.200,nw_tos=0,nw_ecn=0,nw_ttl=116,nw_frag=no,icmp_type=0,icmp_code=0
> Megaflow:
> recirc_id=0x130b9b,ct_state=+new-est-rel-rpl-inv+trk,ct_mark=0/0x1,eth,icmp,in_port=3,dl_src=b0:cf:0e:04:3d:fd,dl_dst=fa:16:00:00:00:00/ff:ff:00:00:00:00,nw_dst=194.146.52.200,nw_ttl=117,nw_frag=no
> Datapath actions: drop
>
> ###
>
> Somehow destination address has not been changed and after recirc(0x130b9b)
> ofproto/trace show that packet is dropped.
> I have played with --ct-next param but without success.
>
> Datapath flows:
> [root@network002 /]# ovs-appctl dpctl/dump-flows -m| grep 8.8.8.8 | grep
> 194.146.52.200
> ufid:90e41d7a-0204-46da-88a3-adfb9fb92e61,
> recirc_id(0x130b9a),dp_hash(0/0),skb_priority(0/0),tunnel(tun_id=0xa5,src=10.84.69.33,dst=10.84.69.24,ttl=64/0,tp_src=18711/0,tp_dst=6081/0,geneve({class=0x102/0,type=0x80/0,len=4/0,0x30002/0}),flags(-df+csum+key)),in_port(genev_sys_6081),skb_mark(0/0),ct_state(0x62/0x3f),ct_zone(0xab/0),ct_mark(0/0x1),ct_label(0/0),ct_tuple4(src=10.44.1.174/0.0.0.0,dst=8.8.8.8/0.0.0.0,proto=1/0,tp_src=8/0,tp_dst=0/0),packet_type(ns=0,id=0),eth(src=fa:16:3e:80:da:f6,dst=00:00:5e:00:01:01),eth_type(0x0800),ipv4(src=194.146.52.200/254.0.0.0,dst=8.8.8.8/128.0.0.0,proto=1/0,tos=0/0,ttl=63/0,frag=no),icmp(type=8/0,code=0/0),
> packets:5658, bytes:554484, used:0.104s, dp:ovs,
> actions:ct_clear,push_vlan(vid=170,pcp=0),p1,
> dp-extra-info:miniflow_bits(10,2)
> ufid:b34d70c8-e45a-4b5b-b9df-4f6f81196c38,
> recirc_id(0),dp_hash(0/0),skb_priority(0/0),in_port(p0),skb_mark(0/0),ct_state(0/0),ct_zone(0/0),ct_mark(0/0),ct_label(0/0),packet_type(ns=0,id=0),eth(src=b0:cf:0e:04:3d:fd,dst=fa:16:3e:80:da:f6),eth_type(0x8100),vlan(vid=170,pcp=0),encap(eth_type(0x0800),ipv4(src=8.8.8.8/254.0.0.0,dst=194.146.52.200,proto=1,tos=0/0,ttl=117,frag=no),icmp(type=0/0xf8,code=0/0)),
> packets:5658, bytes:577116, used:0.104s, dp:ovs,
> actions:pop_vlan,ct(zone=171,nat),recirc(0x130b9b),
> dp-extra-info:miniflow_bits(5,3)
>
> Conntrack:
> [root@network002 /]# ovs-appctl dpctl/dump-conntrack zone=171
> icmp,orig=(src=10.44.1.174,dst=8.8.8.8,id=3714,type=8,code=0),reply=(src=8.8.8.8,dst=194.146.52.200,id=3714,type=0,code=0),zone=171
>
> What I would suspect is for recirc(0x130b9b) destination IP would be changed
> to local one and traffic be pushed to VM.
> Like (changed manually):
>
> [root@network002 /]# ovs-appctl ofproto/trace br-ex
> "recirc_id=0x130b9b,ct_state=new|trk,ct_zone=171,icmp,reg0=0xfa16,reg1=0x3e80daf6,reg9=0x4,reg11=0x97,reg12=0xab,reg14=0x1,metadata=0xa5,in_port=3,vlan_tci=0x0000,dl_src=b0:cf:0e:04:3d:fd,dl_dst=fa:16:3e:80:da:f6,nw_src=8.8.8.8,nw_dst=10.44.1.174,nw_tos=0,nw_ecn=0,nw_ttl=117,nw_frag=no,icmp_type=0,icmp_code=0"
>
> <CUT>
>
> 39. reg13=0/0xffff0000,reg15=0x4,metadata=0xa4, priority 100, cookie
> 0x77f10a22
> set_field:0xa4/0xffffff->tun_id
> set_field:0x4->tun_metadata0
> move:NXM_NX_REG14[0..14]->NXM_NX_TUN_METADATA0[16..30]
> -> NXM_NX_TUN_METADATA0[16..30] is now 0x2
> output:1
> -> output to native tunnel
> -> tunneling to 10.84.69.33 via os_tunnel
> -> tunneling from 56:81:8e:92:af:64 10.84.69.24 to
> 9e:51:6b:8a:4b:76 10.84.69.33
>
> bridge("br-ex")
> ---------------
> 0. priority 0
> NORMAL
> -> forwarding to learned port
> resubmit(,40)
> 40. priority 0
> drop
> pop:NXM_OF_IN_PORT[]
> -> NXM_OF_IN_PORT[] is now 3
>
> Final flow: unchanged
> Megaflow:
> recirc_id=0x132219,ct_state=+new-est-rel-rpl-inv+trk,ct_mark=0/0x1,eth,icmp,in_port=3,dl_src=fa:16:3e:e2:37:bc,dl_dst=fa:16:3e:32:db:00,nw_ecn=0,nw_frag=no
> Datapath actions:
> ct_clear,tnl_push(tnl_port(1),header(size=58,type=5,eth(dst=9e:51:6b:8a:4b:76,src=56:81:8e:92:af:64,dl_type=0x0800),ipv4(src=10.84.69.24,dst=10.84.69.33,proto=17,tos=0,ttl=64,frag=0x4000),udp(s
> rc=0,dst=6081,csum=0xffff),geneve(crit,vni=0xa4,options({class=0x102,type=0x80,len=4,0x20004}))),out_port(6)),push_vlan(vid=140,pcp=0),4
>
> ###
>
> What I am missing here?
Hi, Krzystzof. Unfortunately, trace can only mimic behavior of the connection
tracking, but it doesn't use the actual conntrack states in the datapath.
The main reason is that passing packets through connection tracking will change
the state of the connection tracker, potentially breaking the real traffic flow.
You can use ct-next flag to give trace a hint on what should be a resulted ct
state, but it doesn't really help with unSNAT case.
The only option here, I think, is for you to modify (unSNAT) the packet yourself
and use ofproto/trace-packet-out with actions=resubmit(,<table-id>) to continue
processing from the table the previous trace stopped. This will not help with
debuggign the conntrack itself, but should be useful if you just want to know
where the packet supposed to go next.
Best regards, Ilya Maximets.
>
> --
> Regards,
>
> Krzystzof Tomaszewski
_______________________________________________
discuss mailing list
disc...@openvswitch.org
https://mail.openvswitch.org/mailman/listinfo/ovs-discuss
