Hi, Just as a follow up, after setting the other_config:broadcast-arps-to-all-routers=false parameter on all external networks, number of packets sent to the ovn-controller went down from around 120-130 p/s to 1-3/s. Most of our ports in external networks are router ports, so this parameter helps a lot.
Thanks Piotr Misiak pon., 17 lut 2025, 16:37 użytkownik Piotr Misiak <piotrmisiak1...@gmail.com> napisał: > Hi, > > On 17.02.2025 08:59, Felix Huettner wrote: > > On Mon, Feb 17, 2025 at 08:33:10AM +0100, Ales Musil via discuss wrote: > >> On Fri, Feb 14, 2025 at 3:57 PM Piotr Misiak via discuss < > >> ovs-discuss@openvswitch.org> wrote: > >> > >>> Hi, > >>> > >> Hi Piotr, > >> > >> thank you for contacting us. > >> > >> > >>> We are running several OpenStack/OVN regions with different sizes. > >>> All of them have external networks connected to the Internet. > >>> We are receiving a lot of packets to non used (non provisioned) > >>> destination IP addresses, I guess some bots scanning Internet. > >>> This creates a lot of ARP requests which cannot be replied, because > >>> those IP addresses are not configured anywhere yet. > >>> > >>> Few days ago we upgraded one of our regions from OVN 22.09 to OVN > >>> 24.03 and basically we suddenly started having critical issues with > >>> DNS resolving on VMs running in the OpenStack. > >>> Generally non of DNS requests were successful, some of them was going > >>> back after 5 minutes, sometimes even after 30 minutes. Yes, minutes > >>> not seconds. > >>> > >> Slightly related, there was recently an improvement to DNS handling > >> where the cache is no longer processed by pinctrl only [0], then > >> later on there was another addition to avoid mutex contention as > >> much as possible [1]. I believe that both of those would help in > >> your case to some extent. > >> > >> > >> > >>> After some debugging we identified problematic OpenFlow flows which > >>> send ARP request packets to ovn-controllers. > >>> Those flows are created because we have around 400 ports in the > >>> external-network and packet flooding flow have to be splitted. > >>> Those flows are installed at the beginning of OF 39 table with > >>> priority 110 which includes 170 resubmits: > >>> > >> Those flows are related to multicast groups, in this case the > "_MC_flood". > >> > >> > >>> cookie=0x28ef9c32, duration=829.596s, table=39, n_packets=117482, > >>> n_bytes=4947460, idle_age=0, hard_age=58, > >>> priority=110,reg6=0x9001,reg15=0x8000,metadata=0xba > >>> > >>> > actions=load:0->NXM_NX_REG6[],load:0x5a3->NXM_NX_REG15[],resubmit(,41),load:0x21af->NXM_NX_REG15[],resubmit(,41),load:0x8f->NXM_NX_REG15[],resubmit(,41),load:0x1374->NXM_NX_REG15[],resubmit(,41),load:0x5f->NXM_NX_REG15[],resubmit(,41),load:0x10b->NXM_NX_REG15[],resubmit(,41),load:0x106->NXM_NX_REG15[],resubmit(,41),load:0x13d9->NXM_NX_REG15[],resubmit(,41),load:0x4d->NXM_NX_REG15[],resubmit(,41),load:0x2202->NXM_NX_REG15[],resubmit(,41),load:0xb4->NXM_NX_REG15[],resubmit(,41),load:0x25ed->NXM_NX_REG15[],resubmit(,41),load:0x1b59->NXM_NX_REG15[],resubmit(,41),load:0x26b2->NXM_NX_REG15[],resubmit(,41),load:0x6a->NXM_NX_REG15[],resubmit(,41) > >>> <<< CUT >>> > >>> > >>> > load:0x169a->NXM_NX_REG15[],resubmit(,41),controller(userdata=00.00.00.1b.00.00.00.00.00.00.90.01.00.00.80.00.27) > >>> > >>> there is also second rule with 170 resubmits with controller() at the > end: > >>> controller(userdata=00.00.00.1b.00.00.00.00.00.00.90.02.00.00.80.00.27) > >>> > >>> and also third rule with smaller number of resubmits without > >>> controller. In total we have around 400 resubmits. > >>> > >>> This was introduced in 24.03 version by this commit: > >>> > >>> > https://github.com/ovn-org/ovn/commit/325c7b203d8bfd12bc1285ad11390c1a55cd6717 > >>> > >>> What we see in the ovn-controller logs: > >>> > >>> 2025-02-12T20:35:41.490Z|10791|pinctrl(ovn_pinctrl0)|DBG|pinctrl > >>> received packet-in | opcode=unrecognized(27)| OF_Table_ID=39| > >>> OF_Cookie_ID=0x28ef9c32| in-port=60| src-mac=4e:15:bc:ac:36:45, > >>> dst-mac=ff:ff:ff:ff:ff:ff| src-ip=A.A.A.A, dst-ip=B.B.B.B > >>> 2025-02-12T20:35:41.500Z|10792|pinctrl(ovn_pinctrl0)|DBG|pinctrl > >>> received packet-in | opcode=unrecognized(27)| OF_Table_ID=39| > >>> OF_Cookie_ID=0x28ef9c32| in-port=65533| src-mac=4e:15:bc:ac:36:45, > >>> dst-mac=ff:ff:ff:ff:ff:ff| src-ip=A.A.A.A, dst-ip=B.B.B.B > >>> > >>> as you can see the same packet is looped thru the ovn-controller > >>> twice. It's because we have 400 ports and this is covered by three > >>> OpenFlow flows. > >>> > >>> The funny thing is that those packets are dropped at the end of > >>> OpenFlow table chain in the datapath. So they kill our ovn-controllers > >>> performance to be finally dropped. > >>> I'm including a small part of packet trace result here: > >>> > >>> 39. reg15=0x8000,metadata=0xba, priority 100, cookie 0x28ef9c32 > >>> set_field:0->reg6 > >>> set_field:0xe8->reg15 > >>> resubmit(,41) > >>> 41. priority 0 > >>> set_field:0->reg0 > >>> set_field:0->reg1 > >>> set_field:0->reg2 > >>> set_field:0->reg3 > >>> set_field:0->reg4 > >>> set_field:0->reg5 > >>> set_field:0->reg6 > >>> set_field:0->reg7 > >>> set_field:0->reg8 > >>> set_field:0->reg9 > >>> resubmit(,42) > >>> 42. metadata=0xba, priority 0, cookie 0x3372823b > >>> resubmit(,43) > >>> 43. metadata=0xba,dl_dst=01:00:00:00:00:00/01:00:00:00:00:00, > >>> priority 110, cookie 0xaabcf4fa > >>> resubmit(,44) > >>> 44. metadata=0xba, priority 0, cookie 0x9b7d541f > >>> resubmit(,45) > >>> 45. metadata=0xba, priority 65535, cookie 0xedb6d3de > >>> resubmit(,46) > >>> 46. metadata=0xba, priority 65535, cookie 0x1dbceae > >>> resubmit(,47) > >>> 47. metadata=0xba, priority 0, cookie 0xc1c2a264 > >>> resubmit(,48) > >>> 48. metadata=0xba, priority 0, cookie 0x640d65ba > >>> resubmit(,49) > >>> 49. metadata=0xba, priority 0, cookie 0x78f2abc0 > >>> resubmit(,50) > >>> 50. metadata=0xba, priority 0, cookie 0x7b63c11c > >>> resubmit(,51) > >>> 51. metadata=0xba,dl_dst=01:00:00:00:00:00/01:00:00:00:00:00, > >>> priority 100, cookie 0xb055fd1c > >>> set_field:0/0x8000000000000000000000000000->xxreg0 > >>> resubmit(,52) > >>> 52. metadata=0xba, priority 0, cookie 0x4dd5d603 > >>> resubmit(,64) > >>> 64. priority 0 > >>> resubmit(,65) > >>> 65. reg15=0xe8,metadata=0xba, priority 100, cookie 0xfab6eb > >>> > >>> > clone(ct_clear,set_field:0->reg11,set_field:0->reg12,set_field:0/0xffff->reg13,set_field:0x25b->reg11,set_field:0x30a->reg12,set_field:0x252->metadata,set_field:0x1->reg14,set_field:0->reg10,set_field:0->reg15,set_field:0->reg0,set_field:0->reg1,set_field:0->reg2,set_field:0->reg3,set_field:0->reg4,set_field:0->reg5,set_field:0->reg6,set_field:0->reg7,set_field:0->reg8,set_field:0->reg9,resubmit(,8)) > >>> ct_clear > >>> set_field:0->reg11 > >>> set_field:0->reg12 > >>> set_field:0/0xffff->reg13 > >>> set_field:0x25b->reg11 > >>> set_field:0x30a->reg12 > >>> set_field:0x252->metadata > >>> set_field:0x1->reg14 > >>> set_field:0->reg10 > >>> set_field:0->reg15 > >>> set_field:0->reg0 > >>> set_field:0->reg1 > >>> set_field:0->reg2 > >>> set_field:0->reg3 > >>> set_field:0->reg4 > >>> set_field:0->reg5 > >>> set_field:0->reg6 > >>> set_field:0->reg7 > >>> set_field:0->reg8 > >>> set_field:0->reg9 > >>> resubmit(,8) > >>> 8. > >>> reg14=0x1,metadata=0x252,dl_dst=01:00:00:00:00:00/01:00:00:00:00:00, > >>> priority 50, cookie 0x33587607 > >>> > >>> > set_field:0xfa163e9f2f460000000000000000/0xffffffffffff0000000000000000->xxreg0 > >>> resubmit(,9) > >>> 9. metadata=0x252, priority 0, cookie 0x671d3d97 > >>> set_field:0x4/0x4->xreg4 > >>> resubmit(,10) > >>> 10. reg9=0x4/0x4,metadata=0x252, priority 100, cookie > 0xd21e0659 > >>> resubmit(,79) > >>> 79. reg0=0x2, priority 0 > >>> drop > >>> resubmit(,11) > >>> 11. arp,metadata=0x252, priority 85, cookie 0xb5758416 > >>> drop > >>> > >>> > >>> What we can do to improve those ARP packets handling to not to send > >>> them to ovn-controllers? > >>> > >> I'm not sure if there is a way to not send them to ovn-controller when > >> the multicast group is large. > >> > >> > >>> Maybe they can be dropped somewhere earlier in the table chain? They > >>> are requesting a MAC address which OVN doesn't know. Why it tries to > >>> flood it to all router ports in the external network? > >>> > >> > >> At this point in the pipeline OVN doesn't know that this IP/MAC is > >> unknown. And because the packet is multicast one OVN basically does > >> what a normal network would do, flood it to all ports on the switch. > > Hi Ales, Hi Piotr, > > > > we had a similar issue in the past. > > However i am not sure if our solution will also work in your case. > > > > What we did is configure the external LS (so the one that does all this > > flooding) with other_config:broadcast-arps-to-all-routers=false. > > This ensures that any arp/nd request that is not handled by the arp > > responder flows is not flooded to LRs. > > In your case that would probably mean it will drop the packets. > > > > Note that this breaks GARPs from the upstream switches since they will > > be dropped too. In our case that is not an issue since we use stable > > virtual mac/ip's. > > > > Let me know if that helps. If not then we will also need to find a > > solution for this on our upcoming 24.03 upgrade :) > > > > Thanks a lot, > > Felix > > > > This is a very helpful hint. First tests show that this will fix our > issue and also issues with 4096 resubmit limit. > > I will let you know if setting > other_config:broadcast-arps-to-all-routers=false on our production > regions lowered number of packets sent to ovn-controller. > > Thank you Felix! > > > >> > >>> Maybe we can implement this "too big" OpenFlow rule in a different way > >>> and loop it inside the fast datapath, if possible? > >>> > >> Unfortunately not, OvS would still try to fit it into a single buffer > >> it doesn't matter if it's one long action or multiple resubmits. > >> Unless there is a action that needs to be executed before > >> continuation e.g. controller action, we would still have the issue > >> that the commit tried to fix. > >> > >> > >> > >>> I also noticed that IPv6 NS packets are processed via ovn-controller. > >>> Why OVS can't create responses inside the fast datapath in a similar > >>> way it creates responses to the ARP requests for known MACs? > >>> > >> This is a known limitation of OvS, there was an attempt to make it > >> work, however it didn't lead anywhere [2]. We should probably try > >> to revisit this. Once there is OvS support we could easily change > >> it in OVN to do it directly as we do for ARP. > >> > >> > >> > >>> This issue had a big influence on our cloud, because the same > >>> ovn-controller thread is responsible for DHCP, DNS interception, IPv6 > >>> NS packets and when they were overloaded all those services were not > >>> working. > >>> > >>> Another thing, quite misleading, are those "opcode=unrecognized(27)" > >>> in the ovn-controller log, which are unrecognized only because I guess > >>> the mentioned commit haven't added new action name mapping somewhere > >>> here: > >>> > >>> > https://github.com/ovn-org/ovn/blob/ed2790153c07a376890f28b0a16bc321e3af016b/lib/actions.c#L5977 > >> > >> Good catch, we might be actually missing more of those looking at it. > >> > >> > >>> To recover our region we disabled the DNS interception and lowered > >>> number of ARP requests by increasing > >>> "net.ipv4.neigh.default.retrans_time_ms" on our upstream gateways. > >>> Those changes lowered number of packets sent to ovn-controllers from > >>> around 500 p/s to 200 p/s and stabilized our region. > >>> Nevertheless this OVN performance issue is still there. > >>> > >> If I may suggest another potential mitigation might be to add > >> stateless ACL that will ensure the ARP packets are dropped before > >> reaching the flood flows. Would that be an option? This would really > >> be just mitigation until we have a proper solution. Speaking about > >> proper solution, given the need for this, the proper solution would > >> probably be CoPP for this controller action so we don't end up with > >> overloaded pinctrl thread. There is a downside to CoPP as we might > >> drop legitimate packets that need flooding. > >> > >> > >>> Thanks for your attention, > >>> Piotr Misiak > >>> _______________________________________________ > >>> discuss mailing list > >>> disc...@openvswitch.org > >>> https://mail.openvswitch.org/mailman/listinfo/ovs-discuss > >>> > >>> > >> Thanks, > >> Ales > >> > >> [0] https://github.com/ovn-org/ovn/commit/817d4e53 > >> [1] https://github.com/ovn-org/ovn/commit/eba60b27 > >> [2] > >> > http://patchwork.ozlabs.org/project/openvswitch/patch/20200928134947.48269-1-fankaixi...@bytedance.com > >> _______________________________________________ > >> discuss mailing list > >> disc...@openvswitch.org > >> https://mail.openvswitch.org/mailman/listinfo/ovs-discuss >
_______________________________________________ discuss mailing list disc...@openvswitch.org https://mail.openvswitch.org/mailman/listinfo/ovs-discuss
