Public bug reported: Hi, I am running openstack ussuri with ovs and DVR routers.
When there are multiple subnets in one network, Neutron does not consider the possibility that the subnets could be connected to different routers. This is a problem when a DVR router is expecting to receive an ARP reply. In OVS br-int, table 3 contains only one rule per network which applies to traffic destined to the DVR MAC. This rule translates the DVR MAC to the MAC of the newest router on the network but does not take into consideration that a network could have multiple subnets connected to different routers. The use case where I am facing this issue is with manila. Manila defines one network object in the service project but each time a user creates a new "Share Network", the Manila service creates a new subnet within the network. So you can end up with many subnets and routers within a network. It is a bit confusing so below are more details taking my use case with manila as an example. Manila has a network called manila_service_network In manila.conf a CIDR and mask is configured and the subnets that will be created by manila service are configured within the CIDR defined and using the mask that is defined. On a user project I create NetworkX/SubnetX (172.20.20.0/24) and connect it to routerX. I also have instanceX on this network. Then I create a Share network. This creates a subnet within network manila_service_network with IP 10.128.16.0/20. Manila_subnet1 (10.128.16.0/20) is connected to routerX which is already connected to SubnetX (172.20.20.0/24). A ShareInstance is created on the manila_subnet1 and has IP 10.128.19.189. It is important that the ShareInstance and InstanceX be located on different computes. Now communication between InstanceX and the ShareInstance should work but it does not. Here's why. InstanceX wants to communicate to the ShareInstance so it sends a packet to it's gateway RouterX. RouterX needs to route the packet to the ShareInstance but it does not have the MAC address in its ARP table. Router X sends an ARP request -> ARP, Request who-has 10.128.19.189 tell 10.128.16.1, length 28 RouterX never receives an ARP reply. I followed the flows in br-int and br-tun. Since traffic is coming from a DVR router, OVS br-tun changes the router's source MAC to the computes's DVR MAC. fa:16:3e:80:4c:3a is the MAC of the router with IP 10.128.16.1 cookie=0x7027c9402a453a34, duration=411942.542s, table=1, n_packets=434, n_bytes=33812, idle_age=589, hard_age=65534, priority=1,dl_vlan=31,dl_src=fa:16:3e:80:4c:3a actions=mod_dl_src:fa:16:3f:67:83:30,resubmit(,2) Then the packet reaches the ARP responder table 21. There is an entry in table 21 for the ShareInstance MAC so it modifies the packet and sends it back to br-int. cookie=0x7027c9402a453a34, duration=11769.612s, table=21, n_packets=23, n_bytes=966, idle_age=2, priority=1,arp,dl_vlan=31,arp_tpa=10.128.19.189 actions=load:0x2->NXM_OF_ARP_OP[],move:NXM_NX_ARP_SHA[]->NXM_NX_ARP_THA[],move:NXM_OF_ARP_SPA[]->NXM_OF_ARP_TPA[],load:0xfa163ee3273f->NXM_NX_ARP_SHA[],load:0xa8013bd->NXM_OF_ARP_SPA[],move:NXM_OF_ETH_SRC[]->NXM_OF_ETH_DST[],mod_dl_src:fa:16:3e:e3:27:3f,IN_PORT But remember that the router source MAC became the DVR MAC and, because of table 21, it is now the destination MAC. This br-int table 1 rules sends us to table 3 because of the destination MAC being the DVR MAC. cookie=0xe728ac45412eb352, duration=4676042.487s, table=0, n_packets=10695122, n_bytes=449195124, idle_age=0, hard_age=65534, priority=5,in_port=2,dl_dst=fa:16:3f:67:83:30 actions=resubmit(,3) In table 3 there is a rule that changes the destination MAC from the DVR MAC to the router MAC based on the VLAN (network). In our case vlan 31. cookie=0xe728ac45412eb352, duration=23642.517s, table=3, n_packets=10626537, n_bytes=446314554, idle_age=0, priority=5,dl_vlan=31,dl_dst=fa:16:3f:67:83:30 actions=mod_dl_dst:fa:16:3e:4d:d0:f9,strip_vlan,output:725 You can see that the mod_dl_dst MAC (fa:16:3e:4d:d0:f9) is not the original source MAC of my router (fa:16:3e:80:4c:3a). Why? Because there are multiple subnets in the network manila_service_network, each connected to a different router. fa:16:3e:4d:d0:f9 belongs to a router connected to Manila_subnet2 (10.128.48.0/20) which is within manila_service_network. This means the ARP reply is sent to the wrong router. All the subnets in manila_service_network use vlan 31 so, by having one rule in table 3 for vlan 31, causes all traffic to be sent to one router (usually the newest). In my use case there are 6 subnets in manila_service_network and the ARP replies for all 6 subnets go to the same router (usually the newest). This means 5 subnets out of 6 are broken. You dont need to use manila to recreate the problem. You need networkA with subnetA and network1 with subnet1, subnet2, subnet3, etc... Connect subnetA and subnet1 to the same router and create a couple instances on subnetA and subnet1 (they need to be on different computes). Then connect subnet2 to a new router and subnet3 to another new router. You should see that the instances on subnetA and subnet1 wont be able to communicate as the traffic will stop on the router. Table 3 is a fairly new addition to neutron/ovs so I believe that the possibility of having multiple subnets in one network was not considered when writing the code. I think it is related to this commit https://review.opendev.org/c/openstack/neutron/+/651905/12/neutron/plugins/ml2/drivers/openvswitch/agent/openflow/native/br_int.py Though it is rare to have multiple subnets in one network, the manila service uses this architecture, making it important that it works correctly. ** Affects: neutron Importance: Undecided Status: New -- You received this bug notification because you are a member of Yahoo! Engineering Team, which is subscribed to neutron. https://bugs.launchpad.net/bugs/1913646 Title: DVR router ARP traffic broken for networks containing multiple subnets Status in neutron: New Bug description: Hi, I am running openstack ussuri with ovs and DVR routers. When there are multiple subnets in one network, Neutron does not consider the possibility that the subnets could be connected to different routers. This is a problem when a DVR router is expecting to receive an ARP reply. In OVS br-int, table 3 contains only one rule per network which applies to traffic destined to the DVR MAC. This rule translates the DVR MAC to the MAC of the newest router on the network but does not take into consideration that a network could have multiple subnets connected to different routers. The use case where I am facing this issue is with manila. Manila defines one network object in the service project but each time a user creates a new "Share Network", the Manila service creates a new subnet within the network. So you can end up with many subnets and routers within a network. It is a bit confusing so below are more details taking my use case with manila as an example. Manila has a network called manila_service_network In manila.conf a CIDR and mask is configured and the subnets that will be created by manila service are configured within the CIDR defined and using the mask that is defined. On a user project I create NetworkX/SubnetX (172.20.20.0/24) and connect it to routerX. I also have instanceX on this network. Then I create a Share network. This creates a subnet within network manila_service_network with IP 10.128.16.0/20. Manila_subnet1 (10.128.16.0/20) is connected to routerX which is already connected to SubnetX (172.20.20.0/24). A ShareInstance is created on the manila_subnet1 and has IP 10.128.19.189. It is important that the ShareInstance and InstanceX be located on different computes. Now communication between InstanceX and the ShareInstance should work but it does not. Here's why. InstanceX wants to communicate to the ShareInstance so it sends a packet to it's gateway RouterX. RouterX needs to route the packet to the ShareInstance but it does not have the MAC address in its ARP table. Router X sends an ARP request -> ARP, Request who-has 10.128.19.189 tell 10.128.16.1, length 28 RouterX never receives an ARP reply. I followed the flows in br-int and br-tun. Since traffic is coming from a DVR router, OVS br-tun changes the router's source MAC to the computes's DVR MAC. fa:16:3e:80:4c:3a is the MAC of the router with IP 10.128.16.1 cookie=0x7027c9402a453a34, duration=411942.542s, table=1, n_packets=434, n_bytes=33812, idle_age=589, hard_age=65534, priority=1,dl_vlan=31,dl_src=fa:16:3e:80:4c:3a actions=mod_dl_src:fa:16:3f:67:83:30,resubmit(,2) Then the packet reaches the ARP responder table 21. There is an entry in table 21 for the ShareInstance MAC so it modifies the packet and sends it back to br-int. cookie=0x7027c9402a453a34, duration=11769.612s, table=21, n_packets=23, n_bytes=966, idle_age=2, priority=1,arp,dl_vlan=31,arp_tpa=10.128.19.189 actions=load:0x2->NXM_OF_ARP_OP[],move:NXM_NX_ARP_SHA[]->NXM_NX_ARP_THA[],move:NXM_OF_ARP_SPA[]->NXM_OF_ARP_TPA[],load:0xfa163ee3273f->NXM_NX_ARP_SHA[],load:0xa8013bd->NXM_OF_ARP_SPA[],move:NXM_OF_ETH_SRC[]->NXM_OF_ETH_DST[],mod_dl_src:fa:16:3e:e3:27:3f,IN_PORT But remember that the router source MAC became the DVR MAC and, because of table 21, it is now the destination MAC. This br-int table 1 rules sends us to table 3 because of the destination MAC being the DVR MAC. cookie=0xe728ac45412eb352, duration=4676042.487s, table=0, n_packets=10695122, n_bytes=449195124, idle_age=0, hard_age=65534, priority=5,in_port=2,dl_dst=fa:16:3f:67:83:30 actions=resubmit(,3) In table 3 there is a rule that changes the destination MAC from the DVR MAC to the router MAC based on the VLAN (network). In our case vlan 31. cookie=0xe728ac45412eb352, duration=23642.517s, table=3, n_packets=10626537, n_bytes=446314554, idle_age=0, priority=5,dl_vlan=31,dl_dst=fa:16:3f:67:83:30 actions=mod_dl_dst:fa:16:3e:4d:d0:f9,strip_vlan,output:725 You can see that the mod_dl_dst MAC (fa:16:3e:4d:d0:f9) is not the original source MAC of my router (fa:16:3e:80:4c:3a). Why? Because there are multiple subnets in the network manila_service_network, each connected to a different router. fa:16:3e:4d:d0:f9 belongs to a router connected to Manila_subnet2 (10.128.48.0/20) which is within manila_service_network. This means the ARP reply is sent to the wrong router. All the subnets in manila_service_network use vlan 31 so, by having one rule in table 3 for vlan 31, causes all traffic to be sent to one router (usually the newest). In my use case there are 6 subnets in manila_service_network and the ARP replies for all 6 subnets go to the same router (usually the newest). This means 5 subnets out of 6 are broken. You dont need to use manila to recreate the problem. You need networkA with subnetA and network1 with subnet1, subnet2, subnet3, etc... Connect subnetA and subnet1 to the same router and create a couple instances on subnetA and subnet1 (they need to be on different computes). Then connect subnet2 to a new router and subnet3 to another new router. You should see that the instances on subnetA and subnet1 wont be able to communicate as the traffic will stop on the router. Table 3 is a fairly new addition to neutron/ovs so I believe that the possibility of having multiple subnets in one network was not considered when writing the code. I think it is related to this commit https://review.opendev.org/c/openstack/neutron/+/651905/12/neutron/plugins/ml2/drivers/openvswitch/agent/openflow/native/br_int.py Though it is rare to have multiple subnets in one network, the manila service uses this architecture, making it important that it works correctly. To manage notifications about this bug go to: https://bugs.launchpad.net/neutron/+bug/1913646/+subscriptions -- Mailing list: https://launchpad.net/~yahoo-eng-team Post to : yahoo-eng-team@lists.launchpad.net Unsubscribe : https://launchpad.net/~yahoo-eng-team More help : https://help.launchpad.net/ListHelp
