agronaught opened a new issue, #14228:
URL: https://github.com/apache/cloudstack/issues/14228
### problem
When the last vif is unplugged from the public bridge, the agent deletes the
bridge and its parent VLAN interface — even though CloudStack never created
them.
BridgeVifDriver.deleteVnetBr() decides whether a bridge is CloudStack-owned
solely by matching its name against a regex:
`Pattern brNameRegex = Pattern.compile("^br(\\S+)-(\\d+)$");
`
This is intended to match bridges the agent created via generateVnetBrName()
("br" + pifName + "-" + vnetId). But the pattern matches any similarly-named
bridge. There is no provenance tracking — the naming convention is the only
ownership test.
A pre-existing, persistently-declared bridge named brbond0-103 (bond bond0,
VLAN 103) matches exactly, so the agent treats it as its own and runs:
`modifyvlan.sh -o delete -v 103 -p bond0 -b brbond0-103 -d true
`
which deletes both interfaces:
```
ip link delete $vlanDev type vlan # → ip link delete bond0.103
ip link set $vlanBr down
ip link delete $vlanBr type bridge # → ip link delete brbond0-103
```
netplan/networkd do not recreate these without netplan apply or a reboot, so
the loss is permanent until manual intervention.
The failure is silent for as long as the agent keeps running. The public NIC
is validated only as a one-time startup precondition in
LibvirtComputingResource.configure():
```
if (pifs.get("public") == null) {
LOGGER.error("Failed to get public nic name");
throw new ConfigurationException("Failed to get public nic name");
}
```
There is no retry and no degraded mode. The running agent never re-checks —
it merely logs, once per minute:
```
WARN [kvm.resource.LibvirtComputingResource] Failed to read the rx_bytes for
brbond0-103
from /sys/class/net/brbond0-103/statistics/rx_bytes
java.io.FileNotFoundException: File
'/sys/class/net/brbond0-103/statistics/rx_bytes' does not exist
```
so the host continues to report healthy. The damage only surfaces at the
next agent restart — in our case 44 hours later — when the host fails to
reconnect.
### versions
ACS 4.22.1.0
Ubuntu 24.04
KVM hypervisor, Ubuntu 24.04, systemd-networkd + netplan.
The Public traffic type uses a persistently declared bridge, created at boot
by netplan — not by CloudStack:
```
# /etc/netplan/50-cloud-init.yaml (abridged)
vlans:
bond0.103:
id: 103
link: bond0
mtu: 1500
bridges:
brbond0-103:
interfaces: [bond0.103]
mtu: 1500
parameters: {stp: false, forward-delay: 0}
```
# agent.properties
`public.network.device=brbond0-103
`
Zone config: Public traffic type has kvmnetworklabel=brbond0-103; public IP
ranges are vlan://103.
OS / ENVIRONMENT
Ubuntu 24.04, kernel 6.11, systemd-networkd/netplan, bonded 802.3ad uplink
### The steps to reproduce the bug
1. On a KVM host, persistently declare (netplan/networkd/ifupdown) a VLAN
interface
and bridge whose name matches ^br(\S+)-(\d+)$ — e.g. bond0.103 enslaved to
brbond0-103.
2. Set public.network.device=brbond0-103 and the zone's Public
kvmnetworklabel=brbond0-103, with a vlan://103 public IP range.
3. Start the agent; create an isolated network so a VR is placed on this
host with
a nic on the public bridge. Confirm brbond0-103 has the VR vif as its
only VM port.
4. Destroy that VR (or restart the network so the VR is recreated
elsewhere), so the
last vif leaves the bridge.
5. Observe: bond0.103 and brbond0-103 are deleted from the running kernel.
6. Restart cloudstack-agent.
EXPECTED RESULTS
The agent does not delete network interfaces it did not create. Bridges and
VLAN
interfaces that are persistently declared by the host's network
configuration are
left in place when the last vif is unplugged.
ACTUAL RESULTS
Step 5 — kernel logs the interfaces being removed (note "unregistering", i.e.
device deletion rather than link-down):
```
kernel: brbond0-103: port 1(bond0.103) entered disabled state
systemd-networkd: bond0.103: Link DOWN
kernel: bond0.103 (unregistering): left promiscuous mode
systemd-networkd: brbond0-103: Link DOWN
```
Step 6 — the agent then refuses to start:
```
ERROR [kvm.resource.LibvirtComputingResource] Failed to get public nic name
ERROR [cloud.agent.AgentShell] Unable to start agent:
javax.naming.ConfigurationException: Failed to get public nic name
at
LibvirtComputingResource.configure(LibvirtComputingResource.java:1495)
at com.cloud.agent.Agent.<init>(Agent.java:239)
...
```
The host stays Disconnected until the interfaces are recreated by hand.
Guest VMs
already running are unaffected (libvirt and the data plane are independent
of the
agent), which is part of why the fault goes unnoticed.
### What to do about it?
Impact is wider than it first appears. The trigger is "the last vif leaves
the public bridge", not "the host is idle" — in our case the host had four VMs
running throughout, none of them on the public bridge. Since only virtual
routers attach to the public network, and VRs are among the most frequently
destroyed/recreated VMs (network restart, offering change, VR upgrade, HA
event), any host that transiently has no VR on the public network is exposed.
The resulting breakage is invisible until the next agent restart, which in
practice means it surfaces during a maintenance window.
Why "just name it differently" is not a workaround. The bridge name used at
plug time is derived, not read from config — createVnetBr() →
generateVnetBrName(_pifs.get(trafficLabel), vNetId) → "br" + "bond0" + "-" +
"103". Renaming the declared bridge causes the agent to create a second bridge
under the constructed name rather than using the renamed one. For a VLAN-tagged
public network there is no configuration that avoids the matching name.
Suggested fixes (in rough order of preference):
Track provenance — only delete bridges the agent itself created (e.g. a
marker, or state recorded at creation time), rather than inferring ownership
from a name.
Failing that, an opt-out agent property, e.g.
network.bridge.externally.managed=brbond0-103,..., consulted by deleteVnetBr()
before deletion.
At minimum, never delete the parent VLAN interface. Even where the bridge is
plausibly CloudStack's, bond0.103 belongs to the host's network configuration.
Two related observations, filed separately to keep this report focused:
modifyvlan.sh sets no MTU when creating interfaces, so an agent-created VLAN
inherits the parent's (9000 on a jumbo-frame bond) rather than the intended
value.
The ls /sys/class/net/<br>/brif "is the bridge in use?" check is not atomic
with the deletion, so a concurrent plug can race it.
Happy to test a patch.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
