On 9/29/20 12:14 PM, Daniel Alvarez Sanchez wrote: > > > On Tue, Sep 29, 2020 at 11:14 AM Krzysztof Klimonda > <[email protected] > <mailto:[email protected]>> wrote: > > On Tue, Sep 29, 2020, at 10:40, Dumitru Ceara wrote: > > On 9/29/20 12:42 AM, Krzysztof Klimonda wrote: > > > Hi Dumitru, > > > > > > This cluster is IPv4-only for now - there are no IPv6 networks > defined at all - overlay or underlay. > > > > > > However, once I increase a number of routers to ~250, a similar > behavior can be observed when I send ARP packets for non-existing > IPv4 addresses. The following warnings will flood ovs-vswitchd.log > for every address not known to OVN when I run `fping -g > 192.168.0.0/16` <http://192.168.0.0/16>: > > > > > > ---8<---8<---8<--- > > > > 2020-09-28T22:26:40.967Z|21996|ofproto_dpif_xlate(handler6)|WARN|over 4096 > resubmit actions on bridge br-int while processing > > arp,in_port=1,vlan_tci=0x0000,dl_src=fa:16:3e:75:38:be,dl_dst=ff:ff:ff:ff:ff:ff,arp_spa=192.168.0.1,arp_tpa=192.168.0.35,arp_op=1,arp_sha=fa:16:3e:75:38:be,arp_tha=00:00:00:00:00:00 > > > ---8<---8<---8<--- > > > > > > This is even a larger concern for me, as some of our clusters > would be exposed to the internet where we can't easily prevent > scanning of an entire IP range. > > > > > > Perhaps this is something that should be handled differently for > traffic coming from external network? Is there any reason why OVN is > not dropping ARP requests and IPv6 ND for IP addresses it knows > nothing about? Or maybe OVN should drop most of BUM traffic on > external network in general? I think all this network is used for is > SNAT and/or SNAT+DNAT for overlay networks. > > > > > > > Ok, so I guess we need a combination of the existing broadcast domain > > limiting options: > > > > 1. send ARP/NS packets only to router ports that own the target IP > address. > > 2. flood IPv6 ND RS packets only to router ports with IPv6 addresses > > configured and ipv6_ra_configs.address_mode set. > > 3. according to the logical switch multicast configuration either > flood > > unkown IP multicast or forward it only to hosts that registered > for the > > IP multicast group. > > 4. drop all other BUM traffic. > > > > From the above, 1 and 3 are already implemented. 2 is what I suggested > > earlier. 4 would probably turn out to be configuration option that > needs > > to be explicitly enabled on the logical switch connected to the > external > > network. > > > > Would this work for you? > > I believe it would work for me, although it may be a good idea to > consult with neutron developers and see if they have any input on that. > > > I think that's a good plan. Implementing 4) via a configuration option > sounds smart. From an OpenStack point of view, I think that as all the > ports are known, we can just have it on by default. > We need to make sure it works for 'edge' cases like virtual ports, load > balancers and subports (ports with a parent port and a tag) but the idea > sounds great to me. > > Thanks folks for the discussion!
Thinking more about it it's probably not OK to drop all other BUM traffic. Instead we should just flood it on all logical ports of a logical switch _except_ router ports. Otherwise we'll be breaking E-W traffic between VIFs connected to the same logical switch. E.g., VM1 and VM2 connected to the same LS and VM1 sending ARP request for VM2's IP. > > > > > > Thanks, > > Dumitru > > > > > -- Krzysztof Klimonda [email protected] > <mailto:[email protected]> On Mon, Sep 28, > > > 2020, at 21:14, Dumitru Ceara wrote: > > >> On 9/28/20 5:33 PM, Krzysztof Klimonda wrote: > > >>> Hi, > > >>> > > >> Hi Krzysztof, > > >> > > >>> We're still doing some scale tests of OpenStack ussuri with > ml2/ovn driver. We've deployed 140 virtualized compute nodes, and > started creating routers that share single external network between > them. Additionally, each router is connected to a private network. > > >>> Previously[1] we hit a problem of too many logical flows being > generated per router connected to the same "external" network - this > put too much stress on ovn-controller and ovs-vswitchd on compute > nodes, and we've applied a patch[2] to limit a number of logical > flows created per router. > > >>> After we dealt with that we've done more testing and created > 200 routers connected to single external network. After that we've > noticed the following logs in ovs-vswitchd.log: > > >>> > > >>> ---8<---8<---8<--- > > >>> > 2020-09-28T11:10:18.938Z|18401|ofproto_dpif_xlate(handler9)|WARN|over 4096 > resubmit actions on bridge br-int while processing > > icmp6,in_port=1,vlan_tci=0x0000,dl_src=fa:16:3e:9b:77:c3,dl_dst=33:33:00:00:00:02,ipv6_src=fe80::f816:3eff:fe9b:77c3,ipv6_dst=ff02::2,ipv6_label=0x2564e,nw_tos=0,nw_ecn=0,nw_ttl=255,icmp_type=133,icmp_code=0 > > >>> ---8<---8<---8<--- > > >>> > > >>> That starts happening after I create ~178 routers connected to > the same external network. > > >>> > > >>> IPv6 RS ICMP packets are coming from the external network - > that's due to the fact that all virtual compute nodes have IPv6 > address on their interface used for the external network and are > trying to discover a gateway. That's by accident, and we can remove > IPv6 address from that interface, however I'm worried that it would > just hide some bigger issue with flows generated by OVN. > > >>> > > >> Is this an IPv4 cluster; are there IPv6 addresses configured on the > > >> logical router ports connected to the external network? > > >> > > >> If there are IPv6 addresses, do the logical router ports > connected to > > >> the external network have > > >> Logical_Router_Port.ipv6_ra_configs.address_mode set? > > >> > > >> If not, we could try to enhance the broadcast domain limiting > code in > > >> OVN [3] to also limit sending router solicitations only to > router ports > > >> with address_mode configured. > > >> > > >> Regards, > > >> Dumitru > > >> > > >> [3] > > >> > > https://github.com/ovn-org/ovn/blob/20a20439219493f27eb222617f045ba54c95ebfc/northd/ovn-northd.c#L6424 > > >> > > >>> software stack: > > >>> > > >>> ovn: 20.06.2 > > >>> ovs: 2.13.1 > > >>> neutron: 16.1.0 > > >>> > > >>> [1] > > http://lists.openstack.org/pipermail/openstack-discuss/2020-September/017370.html > > >>> [2] https://review.opendev.org/#/c/752678/ > > >>> > > >> > > > > > _______________________________________________ > discuss mailing list > [email protected] <mailto:[email protected]> > https://mail.openvswitch.org/mailman/listinfo/ovs-discuss > _______________________________________________ discuss mailing list [email protected] https://mail.openvswitch.org/mailman/listinfo/ovs-discuss
