Hey there,
we are on OVN 23.06.3 + OVS 3.1.2 and are facing an issue with the ACLs. For
some odd reason some UDP Packets are not dropped. I attached all the
information I was able to gather. The attached traces show the Wireguard
connection between two VMs on Port 51871 (src+dst). This connection should not
work as no UDP Ports except 6081 and 8472 are allowed. It still does tho.
This is the SG representation in Neutron:
❯ openstack security group rule list c987f508-a7de-4bb5-9333-849c129a22b8
+--------------------------------------+-------------+-----------+-----------+-------------+-----------+--------------------------------------+----------------------+
| ID | IP Protocol | Ethertype | IP Range | Port Range | Direction | Remote
Security Group | Remote Address Group |
+--------------------------------------+-------------+-----------+-----------+-------------+-----------+--------------------------------------+----------------------+
| 01c904db-57be-437b-9c74-12f0cbad1460 | udp | IPv4 | 0.0.0.0/0 | 8472:8472 |
ingress | c987f508-a7de-4bb5-9333-849c129a22b8 | None |
| 0b2daa78-b12e-4776-ba49-e9994869d8a8 | tcp | IPv4 | 0.0.0.0/0 | 10250:10250 |
ingress | c987f508-a7de-4bb5-9333-849c129a22b8 | None |
| 154b9565-67f8-4f4b-9838-1dbda43134fe | tcp | IPv4 | 0.0.0.0/0 | 22:22 |
ingress | None | None |
| 17c38fc3-4278-457d-8739-6864afbb2526 | None | IPv4 | 0.0.0.0/0 | | egress |
None | None |
| 1d8f5aab-dfe0-4ed2-9272-6a9c1c0c609f | udp | IPv4 | 0.0.0.0/0 | 6081:6081 |
ingress | c987f508-a7de-4bb5-9333-849c129a22b8 | None |
| 1f8a26db-4779-4678-ab2d-0f0cb9eeef9c | tcp | IPv4 | 0.0.0.0/0 | 4244:4244 |
ingress | afee7436-fd26-4d6e-b2bb-fb955ae6fd4f | None |
| 20faa684-8161-4912-b4f3-a3eae04ce901 | tcp | IPv4 | 0.0.0.0/0 | 10250:10250 |
ingress | afee7436-fd26-4d6e-b2bb-fb955ae6fd4f | None |
| 226b8f5f-03d3-43ab-a0c3-9161d9120794 | tcp | IPv4 | 0.0.0.0/0 | 4250:4250 |
ingress | afee7436-fd26-4d6e-b2bb-fb955ae6fd4f | None |
| 30514756-490b-43b5-bbab-81a0a5acc2ad | None | IPv6 | ::/0 | | egress | None |
None |
| 40f33186-4362-427f-bf8c-3f46d7aca525 | tcp | IPv4 | 0.0.0.0/0 | 4250:4250 |
ingress | c987f508-a7de-4bb5-9333-849c129a22b8 | None |
| 438e32b1-c35c-42ba-89bc-a63139c0737c | tcp | IPv4 | 0.0.0.0/0 | 4240:4240 |
ingress | c987f508-a7de-4bb5-9333-849c129a22b8 | None |
| 463611cf-9732-44a2-8f83-4ef54b4b6e0e | tcp | IPv4 | 0.0.0.0/0 | 4244:4244 |
ingress | c987f508-a7de-4bb5-9333-849c129a22b8 | None |
| 6cd71132-c560-4c3b-98c0-03757e91a04c | icmp | IPv4 | 0.0.0.0/0 | | ingress |
None | None |
| 894349a4-bf1b-4c4f-a33f-df5e1c3a5e83 | tcp | IPv4 | 0.0.0.0/0 | 30000:32767 |
ingress | None | None |
| 8f1991f1-6e6d-45e0-81c9-c4730aa27be1 | tcp | IPv4 | 0.0.0.0/0 | 4240:4240 |
ingress | afee7436-fd26-4d6e-b2bb-fb955ae6fd4f | None |
| 9e25f25e-6bfe-48ae-88e0-ac06253199cb | tcp | IPv4 | 0.0.0.0/0 | 9100:9100 |
ingress | c987f508-a7de-4bb5-9333-849c129a22b8 | None |
| cb020375-066d-4547-b268-be75ac1dad34 | udp | IPv4 | 0.0.0.0/0 | 6081:6081 |
ingress | afee7436-fd26-4d6e-b2bb-fb955ae6fd4f | None |
| d419edce-217f-4816-b214-f93a148bc8f6 | tcp | IPv4 | 0.0.0.0/0 | 9100:9100 |
ingress | afee7436-fd26-4d6e-b2bb-fb955ae6fd4f | None |
| e7fbaf7f-45ba-4099-99a1-1a2994670aad | udp | IPv4 | 0.0.0.0/0 | 8472:8472 |
ingress | afee7436-fd26-4d6e-b2bb-fb955ae6fd4f | None |
+--------------------------------------+-------------+-----------+-----------+-------------+-----------+--------------------------------------+----------------------+
Thanks!
Best regards,
Justin Lamp
--
Justin Lamp
Systems Engineer
NETWAYS Managed Services GmbH | Deutschherrnstr. 15-19 | D-90429 Nuernberg
Tel: +49 911 92885-0 | Fax: +49 911 92885-77
CEO: Julian Hein, Bernd Erk, Sebastian Saemann | AG Nuernberg HRB25207
https://www.netways.de | justin.l...@netways.de
** stackconf 2024 - June | Berlin - https://stackconf.eu **
** OSMC 2024 - November | Nuremberg - https://osmc.de **
** NETWAYS Web Services - https://nws.netways.de **
** NETWAYS Trainings - https://netways.de/trainings **
###
### ACLs of egressport
###
root@ovn-db1:~# ovn-nbctl acl-list pg_c987f508_a7de_4bb5_9333_849c129a22b8
from-lport 1002 (inport == @pg_c987f508_a7de_4bb5_9333_849c129a22b8 && ip4) allow-related
from-lport 1002 (inport == @pg_c987f508_a7de_4bb5_9333_849c129a22b8 && ip6) allow-related
to-lport 1002 (outport == @pg_c987f508_a7de_4bb5_9333_849c129a22b8 && ip4 && icmp4) allow-related
to-lport 1002 (outport == @pg_c987f508_a7de_4bb5_9333_849c129a22b8 && ip4 && ip4.src == $pg_afee7436_fd26_4d6e_b2bb_fb955ae6fd4f_ip4 && tcp && tcp.dst == 10250) allow-related
to-lport 1002 (outport == @pg_c987f508_a7de_4bb5_9333_849c129a22b8 && ip4 && ip4.src == $pg_afee7436_fd26_4d6e_b2bb_fb955ae6fd4f_ip4 && tcp && tcp.dst == 4240) allow-related
to-lport 1002 (outport == @pg_c987f508_a7de_4bb5_9333_849c129a22b8 && ip4 && ip4.src == $pg_afee7436_fd26_4d6e_b2bb_fb955ae6fd4f_ip4 && tcp && tcp.dst == 4244) allow-related
to-lport 1002 (outport == @pg_c987f508_a7de_4bb5_9333_849c129a22b8 && ip4 && ip4.src == $pg_afee7436_fd26_4d6e_b2bb_fb955ae6fd4f_ip4 && tcp && tcp.dst == 4250) allow-related
to-lport 1002 (outport == @pg_c987f508_a7de_4bb5_9333_849c129a22b8 && ip4 && ip4.src == $pg_afee7436_fd26_4d6e_b2bb_fb955ae6fd4f_ip4 && tcp && tcp.dst == 9100) allow-related
to-lport 1002 (outport == @pg_c987f508_a7de_4bb5_9333_849c129a22b8 && ip4 && ip4.src == $pg_afee7436_fd26_4d6e_b2bb_fb955ae6fd4f_ip4 && udp && udp.dst == 6081) allow-related
to-lport 1002 (outport == @pg_c987f508_a7de_4bb5_9333_849c129a22b8 && ip4 && ip4.src == $pg_afee7436_fd26_4d6e_b2bb_fb955ae6fd4f_ip4 && udp && udp.dst == 8472) allow-related
to-lport 1002 (outport == @pg_c987f508_a7de_4bb5_9333_849c129a22b8 && ip4 && ip4.src == $pg_c987f508_a7de_4bb5_9333_849c129a22b8_ip4 && tcp && tcp.dst == 10250) allow-related
to-lport 1002 (outport == @pg_c987f508_a7de_4bb5_9333_849c129a22b8 && ip4 && ip4.src == $pg_c987f508_a7de_4bb5_9333_849c129a22b8_ip4 && tcp && tcp.dst == 4240) allow-related
to-lport 1002 (outport == @pg_c987f508_a7de_4bb5_9333_849c129a22b8 && ip4 && ip4.src == $pg_c987f508_a7de_4bb5_9333_849c129a22b8_ip4 && tcp && tcp.dst == 4244) allow-related
to-lport 1002 (outport == @pg_c987f508_a7de_4bb5_9333_849c129a22b8 && ip4 && ip4.src == $pg_c987f508_a7de_4bb5_9333_849c129a22b8_ip4 && tcp && tcp.dst == 4250) allow-related
to-lport 1002 (outport == @pg_c987f508_a7de_4bb5_9333_849c129a22b8 && ip4 && ip4.src == $pg_c987f508_a7de_4bb5_9333_849c129a22b8_ip4 && tcp && tcp.dst == 9100) allow-related
to-lport 1002 (outport == @pg_c987f508_a7de_4bb5_9333_849c129a22b8 && ip4 && ip4.src == $pg_c987f508_a7de_4bb5_9333_849c129a22b8_ip4 && udp && udp.dst == 6081) allow-related
to-lport 1002 (outport == @pg_c987f508_a7de_4bb5_9333_849c129a22b8 && ip4 && ip4.src == $pg_c987f508_a7de_4bb5_9333_849c129a22b8_ip4 && udp && udp.dst == 8472) allow-related
to-lport 1002 (outport == @pg_c987f508_a7de_4bb5_9333_849c129a22b8 && ip4 && tcp && tcp.dst == 22) allow-related
to-lport 1002 (outport == @pg_c987f508_a7de_4bb5_9333_849c129a22b8 && ip4 && tcp && tcp.dst >= 30000 && tcp.dst <= 32767) allow-related
###
### Trace of UDP packet between nodes
###
root@ovn-db1:~# ovn-trace --ct new 'inport == "20d73b8e-948b-40bd-be4c-27a2034b9bfc" && eth.src == fa:16:3e:14:b3:54 && eth.dst == fa:16:3e:12:8a:91 && ip4.src == 10.0.0.129 && ip4.dst == 10.0.0.212 && ip.ttl == 64 && udp.src == 51871 && udp.dst == 51871'
# udp,reg14=0x3,vlan_tci=0x0000,dl_src=fa:16:3e:14:b3:54,dl_dst=fa:16:3e:12:8a:91,nw_src=10.0.0.129,nw_dst=10.0.0.212,nw_tos=0,nw_ecn=0,nw_ttl=64,nw_frag=no,tp_src=51871,tp_dst=51871
ingress(dp="cl-125-01", inport="cl-125-01-akszmvbdeskq-kube_masters-6xw76efzt362-0-x7nvkcc3yl2r-kube_master_eth0-kxmkywlwmecz")
-------------------------------------------------------------------------------------------------------------------------------
0. ls_in_check_port_sec (northd.c:8583): 1, priority 50, uuid 5525c19b
reg0[15] = check_in_port_sec();
next;
4. ls_in_pre_acl (northd.c:5991): ip, priority 100, uuid b89b125f
reg0[0] = 1;
next;
6. ls_in_pre_stateful (northd.c:6209): reg0[0] == 1, priority 100, uuid 7283ebdc
ct_next;
ct_next(ct_state=new|trk)
-------------------------
7. ls_in_acl_hint (northd.c:6259): ct.new && !ct.est, priority 7, uuid f671789e
reg0[7] = 1;
reg0[9] = 1;
next;
8. ls_in_acl_eval (northd.c:6493): reg0[7] == 1 && (inport == @pg_afee7436_fd26_4d6e_b2bb_fb955ae6fd4f && ip4), priority 2002, uuid b772f47d
reg8[16] = 1;
reg0[1] = 1;
next;
9. ls_in_acl_action (northd.c:6727): reg8[16] == 1, priority 1000, uuid 98d0e052
reg8[16] = 0;
reg8[17] = 0;
reg8[18] = 0;
next;
19. ls_in_acl_after_lb_action (northd.c:6753): 1, priority 0, uuid e9287a54
reg8[16] = 0;
reg8[17] = 0;
reg8[18] = 0;
next;
20. ls_in_stateful (northd.c:7760): reg0[1] == 1 && reg0[13] == 0, priority 100, uuid 421b2df2
ct_commit { ct_mark.blocked = 0; };
next;
27. ls_in_l2_lkup (northd.c:9329): eth.dst == fa:16:3e:12:8a:91, priority 50, uuid d3836ad9
outport = "cl-125-01-akszmvbdeskq-kube_minions-kub3mzovvdoi-1-ekvovdtlevli-kube_minion_eth0-px7gdvfoomje";
output;
egress(dp="cl-125-01", inport="cl-125-01-akszmvbdeskq-kube_masters-6xw76efzt362-0-x7nvkcc3yl2r-kube_master_eth0-kxmkywlwmecz", outport="cl-125-01-akszmvbdeskq-kube_minions-kub3mzovvdoi-1-ekvovdtlevli-kube_minion_eth0-px7gdvfoomje")
---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------
0. ls_out_pre_acl (northd.c:5993): ip, priority 100, uuid 1d41af08
reg0[0] = 1;
next;
2. ls_out_pre_stateful (northd.c:5964): reg0[0] == 1, priority 100, uuid 20a13a99
ct_next;
ct_next(ct_state=est|trk /* default (use --ct to customize) */)
---------------------------------------------------------------
3. ls_out_acl_hint (northd.c:6297): !ct.new && ct.est && !ct.rpl && ct_mark.blocked == 0, priority 4, uuid 68ebedc6
reg0[8] = 1;
reg0[10] = 1;
next;
4. ls_out_acl_eval (northd.c:6553): reg0[10] == 1 && (outport == @neutron_pg_drop && ip), priority 2001, uuid 2111930c
reg8[17] = 1;
ct_commit { ct_mark.blocked = 1; };
next;
5. ls_out_acl_action (northd.c:6731): reg8[17] == 1, priority 1000, uuid 89a304c0
reg8[16] = 0;
reg8[17] = 0;
reg8[18] = 0;
###
### Logical Flows called in trace
###
root@ovn-db1:~# ovn-sbctl lflow-list 2701b3b5-19b0-454c-a0fe-23fc40960719 2111930c
Datapath: "neutron-7bca15eb-72d7-4e91-bf0b-f5af4e14ad5c" aka "cl-125-01" (2701b3b5-19b0-454c-a0fe-23fc40960719) Pipeline: egress
table=4 (ls_out_acl_eval ), priority=2001 , match=(reg0[10] == 1 && (outport == @neutron_pg_drop && ip)), action=(reg8[17] = 1; ct_commit { ct_mark.blocked = 1; }; next;)
root@ovn-db1:~# ovn-sbctl lflow-list 2701b3b5-19b0-454c-a0fe-23fc40960719 89a304c0
Datapath: "neutron-7bca15eb-72d7-4e91-bf0b-f5af4e14ad5c" aka "cl-125-01" (2701b3b5-19b0-454c-a0fe-23fc40960719) Pipeline: egress
table=5 (ls_out_acl_action ), priority=1000 , match=(reg8[17] == 1), action=(reg8[16] = 0; reg8[17] = 0; reg8[18] = 0; /* drop */)
###
### Local OpenvSwitch trace
###
root@openstack-hv7:~# openvswitch-vswitchd ovs-appctl ofproto/trace br-int in_port=$(ovs-vsctl get Interface tap20d73b8e-94 ofport) $flow
Flow: udp,in_port=783,vlan_tci=0x0000,dl_src=fa:16:3e:14:b3:54,dl_dst=fa:16:3e:12:8a:91,nw_src=10.0.0.129,nw_dst=10.0.0.212,nw_tos=0,nw_ecn=0,nw_ttl=64,nw_frag=no,tp_src=51871,tp_dst=51871
bridge("br-int")
----------------
0. in_port=783, priority 100, cookie 0x935a4311
set_field:0x4df->reg13
set_field:0x4dd->reg11
set_field:0x4de->reg12
set_field:0x254->metadata
set_field:0x3->reg14
resubmit(,8)
8. metadata=0x254, priority 50, cookie 0x5525c19b
set_field:0/0x1000->reg10
resubmit(,73)
73. ip,reg14=0x3,metadata=0x254,dl_src=fa:16:3e:14:b3:54,nw_src=10.0.0.129, priority 90, cookie 0x935a4311
set_field:0/0x1000->reg10
move:NXM_NX_REG10[12]->NXM_NX_XXREG0[111]
-> NXM_NX_XXREG0[111] is now 0
resubmit(,9)
9. metadata=0x254, priority 0, cookie 0x65fa76fb
resubmit(,10)
10. metadata=0x254, priority 0, cookie 0x1ef03db3
resubmit(,11)
11. metadata=0x254, priority 0, cookie 0x6e742c27
resubmit(,12)
12. ip,metadata=0x254, priority 100, cookie 0xb89b125f
set_field:0x1000000000000000000000000/0x1000000000000000000000000->xxreg0
resubmit(,13)
13. metadata=0x254, priority 0, cookie 0x24bbc897
resubmit(,14)
14. ip,reg0=0x1/0x1,metadata=0x254, priority 100, cookie 0x7283ebdc
ct(table=15,zone=NXM_NX_REG13[0..15])
drop
-> A clone of the packet is forked to recirculate. The forked pipeline will be resumed at table 15.
-> Sets the packet to an untracked state, and clears all the conntrack fields.
Final flow: udp,reg0=0x1,reg11=0x4dd,reg12=0x4de,reg13=0x4df,reg14=0x3,metadata=0x254,in_port=783,vlan_tci=0x0000,dl_src=fa:16:3e:14:b3:54,dl_dst=fa:16:3e:12:8a:91,nw_src=10.0.0.129,nw_dst=10.0.0.212,nw_tos=0,nw_ecn=0,nw_ttl=64,nw_frag=no,tp_src=51871,tp_dst=51871
Megaflow: recirc_id=0,eth,udp,in_port=783,dl_src=fa:16:3e:14:b3:54,dl_dst=fa:16:3e:12:8a:91,nw_src=10.0.0.129,nw_frag=no,tp_src=0x8000/0x8000
Datapath actions: ct(zone=1247),recirc(0x20b0e97)
===============================================================================
recirc(0x20b0e97) - resume conntrack with default ct_state=trk|new (use --ct-next to customize)
===============================================================================
Flow: recirc_id=0x20b0e97,ct_state=new|trk,ct_zone=1247,eth,udp,reg0=0x1,reg11=0x4dd,reg12=0x4de,reg13=0x4df,reg14=0x3,metadata=0x254,in_port=783,vlan_tci=0x0000,dl_src=fa:16:3e:14:b3:54,dl_dst=fa:16:3e:12:8a:91,nw_src=10.0.0.129,nw_dst=10.0.0.212,nw_tos=0,nw_ecn=0,nw_ttl=64,nw_frag=no,tp_src=51871,tp_dst=51871
bridge("br-int")
----------------
thaw
Resuming from table 15
15. ct_state=+new-est+trk,metadata=0x254, priority 7, cookie 0xf671789e
set_field:0x80000000000000000000000000/0x80000000000000000000000000->xxreg0
set_field:0x200000000000000000000000000/0x200000000000000000000000000->xxreg0
resubmit(,16)
16. ip,reg0=0x80/0x80,reg14=0x3,metadata=0x254, priority 2002, cookie 0xb772f47d
set_field:0x1000000000000/0x1000000000000->xreg4
set_field:0x2000000000000000000000000/0x2000000000000000000000000->xxreg0
resubmit(,17)
17. reg8=0x10000/0x10000,metadata=0x254, priority 1000, cookie 0x98d0e052
set_field:0/0x1000000000000->xreg4
set_field:0/0x2000000000000->xreg4
set_field:0/0x4000000000000->xreg4
resubmit(,18)
18. metadata=0x254, priority 0, cookie 0x18d44fe9
resubmit(,19)
19. metadata=0x254, priority 0, cookie 0x22673cf0
resubmit(,20)
20. metadata=0x254, priority 0, cookie 0xc942bfa7
resubmit(,21)
21. metadata=0x254, priority 0, cookie 0xa17b9eeb
resubmit(,22)
22. metadata=0x254, priority 0, cookie 0xd92dc8e1
resubmit(,23)
23. metadata=0x254, priority 0, cookie 0xa3e28774
resubmit(,24)
24. metadata=0x254, priority 0, cookie 0x29883a85
resubmit(,25)
25. metadata=0x254, priority 0, cookie 0x811dae8d
resubmit(,26)
26. metadata=0x254, priority 0, cookie 0x81ed061f
resubmit(,27)
27. metadata=0x254, priority 0, cookie 0xe9287a54
set_field:0/0x1000000000000->xreg4
set_field:0/0x2000000000000->xreg4
set_field:0/0x4000000000000->xreg4
resubmit(,28)
28. ip,reg0=0x2/0x2002,metadata=0x254, priority 100, cookie 0x421b2df2
ct(commit,zone=NXM_NX_REG13[0..15],nat(src),exec(set_field:0/0x1->ct_mark))
nat(src)
set_field:0/0x1->ct_mark
-> Sets the packet to an untracked state, and clears all the conntrack fields.
resubmit(,29)
29. metadata=0x254, priority 0, cookie 0xa6ed0580
resubmit(,30)
30. metadata=0x254, priority 0, cookie 0xc2012a19
resubmit(,31)
31. metadata=0x254, priority 0, cookie 0xcf7f0fbe
resubmit(,32)
32. metadata=0x254, priority 0, cookie 0x8479ba93
resubmit(,33)
33. metadata=0x254, priority 0, cookie 0xba0c1dbd
resubmit(,34)
34. metadata=0x254, priority 0, cookie 0x35a74cb5
resubmit(,35)
35. metadata=0x254,dl_dst=fa:16:3e:12:8a:91, priority 50, cookie 0xd3836ad9
set_field:0x5->reg15
resubmit(,37)
37. priority 0
resubmit(,39)
39. reg15=0x5,metadata=0x254, priority 100, cookie 0xee93fa70
set_field:0x254/0xffffff->tun_id
set_field:0x5->tun_metadata0
move:NXM_NX_REG14[0..14]->NXM_NX_TUN_METADATA0[16..30]
-> NXM_NX_TUN_METADATA0[16..30] is now 0x3
output:12
-> output to kernel tunnel
resubmit(,40)
40. priority 0
drop
Final flow: recirc_id=0x20b0e97,eth,udp,reg0=0x283,reg11=0x4dd,reg12=0x4de,reg13=0x4df,reg14=0x3,reg15=0x5,tun_id=0x254,metadata=0x254,in_port=783,vlan_tci=0x0000,dl_src=fa:16:3e:14:b3:54,dl_dst=fa:16:3e:12:8a:91,nw_src=10.0.0.129,nw_dst=10.0.0.212,nw_tos=0,nw_ecn=0,nw_ttl=64,nw_frag=no,tp_src=51871,tp_dst=51871
Megaflow: recirc_id=0x20b0e97,ct_state=+new-est-rel-rpl-inv+trk,ct_mark=0/0x1,eth,ip,in_port=783,dl_src=fa:16:3e:14:b3:54,dl_dst=fa:16:3e:12:8a:91,nw_dst=10.0.0.128/25,nw_ecn=0,nw_frag=no
Datapath actions: ct(commit,zone=1247,mark=0/0x1,nat(src)),set(tunnel(tun_id=0x254,dst=10.77.2.57,ttl=64,tp_dst=6081,geneve({class=0x102,type=0x80,len=4,0x30005}),flags(df|csum|key))),3
###
### OpenFlow rule that outputs packets
###
root@openstack-hv7:~# ovs-ofctl dump-flows br-int cookie=0xee93fa70/-1
cookie=0xee93fa70, duration=1185930.217s, table=39, n_packets=4729657, n_bytes=3691075440, priority=100,reg15=0x5,metadata=0x254 actions=load:0x254->NXM_NX_TUN_ID[0..23],set_field:0x5->tun_metadata0,move:NXM_NX_REG14[0..14]->NXM_NX_TUN_METADATA0[16..30],output:"ovn-net-op-e",resubmit(,40)
_______________________________________________
discuss mailing list
disc...@openvswitch.org
https://mail.openvswitch.org/mailman/listinfo/ovs-discuss
