** Description changed:
[ Impact ]
- ovn 26.03.0 (Ubuntu Resolute 26.03.0-2, Ubuntu Cloud Archive gazpacho
- 26.03.0-2~cloud0) drops ARP requests that enter a logical switch through
- its localnet port when the switch also has a distributed gateway router
- port. In OpenStack terms this happens on any provider network that is
- used as a Neutron router external gateway.
+ ovn 26.03.0 (Resolute 26.03.0-2, UCA gazpacho 26.03.0-2~cloud0) includes
+ upstream commit 1b4058b9 ("northd: Process external arps on ha chassis").
+ On a provider network that is used as a router external gateway, it drops
+ ARP requests arriving from the localnet port on every non-gateway
+ chassis, so VMs on that network become unreachable from the physical
+ network and from VMs on other chassis.
- Upstream commit 1b4058b9 ("northd: Process external arps on ha chassis"),
- which first shipped in v26.03.0 (it is not in v25.09.0), installs these
- logical flows on such a switch:
+ Upstream reverted this commit and its 4 follow-ups in v26.03.2:
+ - 78659feb Revert "Process external arps on ha chassis"
+ - edf1f564 Revert "Fix receiving ARP reply on non-resident chassis"
+ - 759ec0da Revert "Fix for distributed dnat_and_snat ARP resolution"
+ - b218c635 Revert "Make sure we process external ARPs even on GW routers"
+ - d93cb42a Revert "Restrict external ARP request to logical_ip for
dnat_and_snat"
- ls_in_check_port_sec priority 75: arp.op == 1 && inport == "provnet-<id>"
-> reg0[22] = 1
- ls_in_apply_port_sec priority 85: reg0[22] == 1 && arp.tpa == <FIP> &&
is_chassis_resident("<fip-port>") -> next
- ls_in_apply_port_sec priority 75: reg0[22] == 1 &&
is_chassis_resident("cr-lrp-<id>") -> next
- ls_in_apply_port_sec priority 70: reg0[22] == 1 -> drop
+ v26.03.2 in turn contains 579abb5e ("northd: Fix HM reply lflow for
+ type=external backends on localnet LS."), which was reverted in v26.03.3
+ (ca33b0da) because it can cause a packet storm.
- On every chassis that is not the gateway chassis, an ARP request coming in
- from the localnet port matches the priority 70 drop. As a result, a VM
- plugged directly into the provider network on a non-gateway compute node
- never receives ARP requests for its own IP from the physical network or
- from VMs on other chassis.
-
- The failure is limited to ARP requests (op=1) that arrive through
- localnet. ARP requests sent by the VM, ARP replies (op=2) and normal IP
- traffic are forwarded correctly. The VM port's port_security setting has
- no effect: VMs with port security enabled are dropped as well.
-
- User-visible effects:
-
- * A VM on the provider network becomes unreachable from outside every
- time the peer's ARP cache entry for it expires, until the VM itself
- sends traffic that refreshes the entry.
-
- * L2 VIPs that only answer ARP requests (for example Cilium L2
- announcements or MetalLB in L2 mode) running inside such VMs are
- unreachable permanently.
-
- * VIP tools that send periodic gratuitous ARP (keepalived, kube-vip)
- keep working.
-
- Logical switches without a router port are not affected. In one
- deployment with 5 provider logical switches, the 3 switches that have a
- router port had the drop flow and the 2 without one did not.
-
- Workaround: send a gratuitous ARP in reply form from the VM
- ("arping -U -P -i <iface> -S <ip> <ip>"). This restores reachability
- until the peers' ARP cache entries expire again.
-
- Fix: upstream reverted 1b4058b9 together with the 4 related commits that
- came with it. The reverts are in v26.03.2 and v26.09.0 (v26.03.1 does not
- include them). v26.03.2 also adds "northd: Enable ARP/ND responder for
- localnet-sourced requests" and the test "ARP to VIF port on LS with
- localnet port". This bug requests an update of ovn in Resolute and in the
- gazpacho Ubuntu Cloud Archive to the 26.03.2 (or later 26.03.x) upstream
- point release.
+ This bug requests updating ovn to the 26.03.3 upstream point release.
[ Test Plan ]
- Minimal topology: one provider (localnet) network, one Neutron router
- whose external gateway is on that network, and one VM on that network on
- a compute node that is not the router's gateway chassis.
-
- Example values used below: provider subnet 192.0.2.0/24, VM IP
- 192.0.2.10, external host 192.0.2.1 connected to the same physical
- network, compute nodes compute-a (VM) and gateway node gw-a.
-
- 1. Create the topology:
-
- openstack network create --external --provider-network-type flat \
- --provider-physical-network physnet1 ext-net
- openstack subnet create --network ext-net --subnet-range 192.0.2.0/24 \
- --gateway 192.0.2.1 ext-subnet
- openstack router create r1
- openstack router set --external-gateway ext-net r1
- openstack server create --network ext-net --flavor <flavor> \
- --image <image> --availability-zone nova:compute-a vm1
-
- Confirm that compute-a is not the gateway chassis of the router port:
-
- ovn-nbctl lrp-get-gateway-chassis lrp-<router-gw-port-id>
-
- 2. Check (a), logical flows:
-
- ovn-sbctl lflow-list neutron-<network-id> | grep 'reg0\[22\]'
-
- Affected (26.03.0): the four flows listed in [ Impact ] are present,
- including "priority=70, match=(reg0[22] == 1), action=(drop;)" in
- ls_in_apply_port_sec.
-
- 3. Check (b), datapath behaviour:
-
- From the external host:
-
- arping -c 3 -I <iface> 192.0.2.10
-
- Affected (26.03.0): no replies.
-
- On compute-a, trace an ARP request entering br-int from the localnet
- patch port. Use the br-int side of the patch pair,
- patch-br-int-to-provnet-<id>. Its peer, patch-provnet-<id>-to-br-int,
- is on the provider bridge; used as in_port on br-int it only hits the
- table 0 default drop on any version. For a VLAN provider network, add
- dl_vlan=<tag>:
-
- ovs-vsctl get Interface patch-br-int-to-provnet-<id> ofport
- ovs-appctl ofproto/trace br-int \
-
in_port=<ofport>,arp,arp_op=1,dl_src=<ext-mac>,dl_dst=ff:ff:ff:ff:ff:ff,arp_sha=<ext-mac>,arp_spa=0:00:00:00,arp_tpa=192.0.2.10
-
- Affected (26.03.0) output, abbreviated:
-
- 8. arp,reg14=0x1,metadata=0x5,arp_op=1, priority 75 -> reg0[22]=1
- 9. reg0=0x400000/0x400000,metadata=0x5, priority 70 -> drop
-
- The same trace with arp_op=2 reaches the VM's tap interface.
- The drop flow counter on compute-a keeps increasing:
-
- ovs-ofctl dump-flows br-int table=9 | grep 'reg0=0x400000/0x400000'
- cookie=..., table=9, n_packets=57988,
priority=70,reg0=0x400000/0x400000,metadata=0x5 actions=drop
-
- The same happens between VMs: when a VM on another compute node sends
- "who-has 192.0.2.10", the request reaches the uplink of compute-a but
- is never delivered to vm1's tap interface.
-
- 4. Upgrade ovn on all nodes to the proposed package and repeat 2 and 3.
- On deployments with VLAN provider networks created by Neutron, do the
- tag_request backfill described in [ Where problems could occur ]
- first.
-
- Expected (fixed):
-
- - The "reg0[22] == 1 -> drop" flow is no longer installed.
- - arping from the external host gets replies.
- - The ofproto/trace for arp_op=1 is no longer dropped:
- * If vm1's port has known addresses (port security enabled), the
- ARP responder added in 26.03.2 answers on compute-a. The datapath
- actions rewrite the packet to an ARP reply (op=2) with vm1's MAC
- and send it back out the uplink; the request itself does not
- reach vm1.
- * If vm1's port has the "unknown" address (port security disabled),
- the request is delivered to vm1's tap interface and vm1 replies.
- * On a chassis where vm1 is not bound, no reply is generated.
- - ARP between VMs on different non-gateway compute nodes works with
- port security both enabled and disabled.
-
- 5. Regression testing: run the usual OpenStack QA process for the UCA
- (deployment plus Tempest smoke tests), including floating IP and
- router gateway connectivity on a provider network. On VLAN provider
- networks, also confirm that every localnet port keeps its tag after
- the upgrade:
-
- ovn-nbctl --columns=name,tag,tag_request find Logical_Switch_Port
- type=localnet
+ 1. Provider network with a Neutron router external gateway on it, and a
+ VM on that network on a compute node that is not the gateway chassis.
+ 2. On 26.03.0, check that the drop flow exists:
+ ovn-sbctl lflow-list | grep 'reg0\[22\] == 1'
+ and that arping to the VM from the physical network gets no reply.
+ 3. Upgrade to the proposed package: the flow is gone and arping gets
+ replies.
+ 4. Usual OpenStack QA for the UCA (deployment plus Tempest smoke tests).
[ Where problems could occur ]
- * This is a minor upstream update from 26.03.0 to 26.03.2, so it brings
- in all other fixes in 26.03.1 and 26.03.2, not only the reverts. Any
- of these could change logical flow generation in northd or flow
- installation in ovn-controller. A regression would show up as lost or
- misrouted traffic on logical switches and routers after the update.
+ * This is a minor upstream update, so it brings in all fixes in
+ 26.03.1, 26.03.2 and 26.03.3. A regression would show up as lost or
+ misrouted traffic on logical switches and routers.
- * 26.03.2 contains "northd: Clear stale LSP tags on tag_request
- port directly into tag and left tag_request empty (LP: #2155789).
- The Neutron fix makes it write tag_request instead, but only for
- ports it creates or updates afterwards; existing localnet ports are
- not backfilled, even after Neutron itself is upgraded.
+ * 26.03.2 and later include cbb71611 ("northd: Clear stale LSP tags on
+ tag_request removal"). Neutron deployments whose VLAN localnet ports
+ only have tag set (see LP: #2155789) lose the VLAN tag after the
+ upgrade. Set tag_request to the current tag on those ports before
+ upgrading.
- On such deployments, upgrading ovn-northd to 26.03.2 removes the VLAN
- tag from the localnet port of every existing VLAN provider network,
- and traffic on those networks stops. Downgrading ovn does not restore
- it, because the tag is removed from the NB database. This was
- reproduced on a test deployment: all VLAN provider networks created
- before the Neutron fix lost connectivity, and the one created after
- it was unaffected.
+ [ Other Info ]
- Before upgrading, set tag_request to the current tag on every
- localnet port that has a tag and no tag_request:
-
- ovn-nbctl --columns=name,tag,tag_request find Logical_Switch_Port
type=localnet
- ovn-nbctl set Logical_Switch_Port <name> tag_request=<tag>
-
- With this backfill done first, the update kept all tags and
- connectivity on the test deployment. Flat provider networks have no
- tag and are not affected. This upgrade step should be documented
- for users of the updated package.
-
- * The five reverted commits restore the 25.09 behaviour for ARP requests
- that arrive from a localnet port on a switch with a distributed
- gateway port. Deployments that came to depend on the 26.03.0 behaviour
- for router gateway or dnat_and_snat (floating IP) addresses would see
- ARP handling for those addresses change after the update.
-
- * 26.03.2 adds a new ARP/ND responder for requests that come from a
- localnet port. Its logical flow only replies when the target logical
- switch port is resident on the local chassis (is_chassis_resident), so
- a port that is not bound to any chassis (status DOWN) but has the
- requested IP reserved gets no reply. This was checked on a test
- deployment: OVN did not answer for an unbound Neutron port with a
- fixed IP on the provider network, and an L2 VIP using that IP inside
- a VM kept working. Deployments that reserve IPs this way should still
- confirm the same behaviour.
-
- * During the upgrade, ovn-northd/NB/SB and ovn-controller run different
- versions for a short time. Upgrade ordering should follow the usual
- OVN guidance (ovn-controller first, then the central components).
-
- [ Other Info ]
-
- * Affected packages:
- - Ubuntu Resolute (26.04): ovn 26.03.0-2
- - Ubuntu Cloud Archive gazpacho (noble): ovn 26.03.0-2~cloud0
-
- * Not affected: v25.09.0 (does not contain 1b4058b9).
-
- * Cause:
- https://github.com/ovn-org/ovn/commit/1b4058b9
- Patch discussion:
- http://www.mail-archive.com/[email protected]/msg99363.html
-
- * Upstream fix, in v26.03.2 and v26.09.0:
- - 78659feb Revert "Process external arps on ha chassis"
- - edf1f564 Revert "Fix receiving ARP reply on non-resident chassis"
- - 759ec0da Revert "Fix for distributed dnat_and_snat ARP resolution"
- - b218c635 Revert "Make sure we process external ARPs even on GW routers"
- - d93cb42a Revert "Restrict external ARP request to logical_ip for
dnat_and_snat"
- Also in v26.03.2: "northd: Enable ARP/ND responder for localnet-sourced
- requests" and the test "ARP to VIF port on LS with localnet port".
- v26.03.1 does not contain the fix.
-
- * Verified on a test deployment with ovn 26.03.2 built from upstream
- source: with the topology above (VLAN provider network, router
- gateway on a separate gateway chassis, VMs on two non-gateway compute
- nodes), the drop flow is gone and ARP from the physical network and
- between VMs on different compute nodes works.
-
- * Related: LP: #2155789 (Neutron writes LSP.tag instead of tag_request),
- https://github.com/ovn-org/ovn/commit/cbb71611
-
- * Changelogs:
- https://www.ovn.org/en/releases/changelog_v26.03.0/
- https://www.ovn.org/en/releases/changelog_v26.03.2/
-
- * OpenStack upgrades do not avoid the problem: kolla stable/2026.2
- still uses OPENSTACK_RELEASE_CODENAME = 'Gazpacho' for its Ubuntu
- images because no noble-updates/hibiscus pocket exists yet, so it
- installs the same ovn 26.03.0 from the gazpacho UCA.
-
- * Other distributions already ship the fix: the CentOS NFV SIG provides
- ovn26.03-26.03.3 and ovn26.09-26.09.0.
-
- * ovn is a supporting package under the OpenStack and Ubuntu Cloud
- Archive SRU exception, so the update is requested as a new upstream
- stable point release.
-
- * Similar previous request: LP: #2168890.
+ * Cause: https://github.com/ovn-org/ovn/commit/1b4058b9
+ * Changelogs:
+ https://www.ovn.org/en/releases/changelog_v26.03.2/
+ https://www.ovn.org/en/releases/changelog_v26.03.3/
+ * ovn is a supporting package under the OpenStack and UCA SRU exception.
+ * Similar previous request: LP: #2168890.
--
You received this bug notification because you are a member of Ubuntu
Bugs, which is subscribed to Ubuntu.
https://bugs.launchpad.net/bugs/2169909
Title:
[SRU] ovn 26.03.0 drops ARP requests from localnet to VMs on non-
gateway chassis when the provider network has a router gateway (fixed
in 26.03.2)
To manage notifications about this bug go to:
https://bugs.launchpad.net/cloud-archive/+bug/2169909/+subscriptions
--
ubuntu-bugs mailing list
[email protected]
https://lists.ubuntu.com/mailman/listinfo/ubuntu-bugs
