andrijapanicsb opened a new issue, #14522:
URL: https://github.com/apache/cloudstack/issues/14522
### problem
=== problem ===
While testing instance boot groups (#13907) on a main nightly build, boot
group starts failed with "Network is not yet fully implemented". The cause
turned out to be unrelated to boot groups. It is the orphaned-network cleanup
added by #12409.
#12409 solves a real problem: a network left in Implementing after a crash
or a lost job could not be used again, and cleaning those up is valuable. The
problem is only in how a "stuck" network is detected. The cleanup also catches
networks that a VM start is implementing at that moment.
#12409 is in no release yet:
- It was merged into 4.20 on 2026-09-09 (milestone 4.20.4) and merged
forward into 4.22 and main on 2026-09-10.
- None of 4.20.3.x, 4.22.1.x or 4.23.0.0 contains it.
- Unless it is fixed, it ships with 4.20.4 and with the next releases cut
from 4.22 and main. The next release from main would ship it together with boot
groups.
**What the code does** (`AsyncJobManagerImpl.cleanupOrphanedNetworks()`,
framework/jobs):
1. It runs in the async-job GC task every 10 seconds (`GC_INTERVAL`).
2. It lists networks in state Implementing. It skips a network whose
**creation** time is within `job.expire.minutes` (default 1440, one day). Once
a network is more than a day old, this check always passes, however briefly the
network has been Implementing.
3. It treats a network as busy only if a pending async job has instance type
`Network` and that network's id. When a VM start or deployment implements a
network, the work runs inside the VM's own job (VmWorkStart, instance type
VirtualMachine), so the cleanup never sees it.
4. It fires `Network.Event.OperationFailed` (Implementing to Shutdown). If
the state machine refuses, it sets Shutdown directly in the database. It does
this without the network lock, which the implementing thread holds
(`NetworkOrchestrator.implementNetwork`, `acquireInLockTable`).
5. Implementing an isolated network includes booting its virtual router.
That took about 36 s here, so a 10-second tick lands inside the implementation
almost every time. When the start then prepares the NICs, the network is in
Shutdown, and the start fails:
`ResourceUnavailableException: Resource [Network:209] is unreachable:
Network is not yet fully implemented`
**Trace** (management-server.log, network id 209):
- 01:19:46-50: the network GC shut the network down (router stopped).
- 01:20:23.709: a VM start took the network's lock and asked the guru and
VirtualRouter to implement it. The router's StartCommand went out at
01:20:24.217.
- 01:20:31.650: `Found orphaned network 209 in Implementing state without
async job. Network created: 2026-10-27T14:32:37, age: 2087 minutes, expiration
threshold: 1440 minutes. Transitioning to Shutdown state.`, followed by
`Successfully transitioned orphaned network 209 to Shutdown state using state
machine`.
- 01:21:00.246: the router start finished, and the lock was released at
01:21:02.030.
- 01:21:02.048: NIC prepare failed with "Network is not yet fully
implemented", and the start failed.
**Why we are confident about the cause:** the test networks were created on
2026-10-27 at 14:32 UTC.
- For the first 24 hours, the same check found networks in Implementing and
skipped them as too young. Nothing failed.
- From 2026-10-28 14:32 on, every cleanup run that found a network in
Implementing marked it as orphaned, and the start implementing it failed. That
was 7 cleanup runs and 17 failed boot group starts.
- One plain startVirtualMachine call also failed its first attempt this way,
and the API's retry passed.
**Impact:**
- **Which networks:** any network older than `job.expire.minutes` (one day
by default, so practically every production network) whenever it has to be
implemented again. Typical cases:
- the first VM start after the network GC shut an idle network down (with
the default `network.gc.*` settings, about 10-20 minutes after its last VM
stopped);
- the first deployment into a network created more than a day ago.
- **Network types:** the cleanup does not filter by type. Any network that
passes through Implementing is exposed; we saw it with isolated networks.
- **Plain start and deploy API:** `VMEntityManagerImpl.deployVirtualMachine`
retries once, which usually passes because the router is up by then. So the
problem is partly hidden: the first attempt fails, a WARN is logged, and the
start takes longer. The retry can be hit too; we saw that once.
- **Paths without a retry fail outright:**
- Instance boot groups (#13907, on main) start VMs through `advanceStart`
once (`InstanceBootGroupManagerImpl` line 201 on main), so the whole group
halts.
- HA restarts (`startVirtualMachineForHA`) also call `advanceStart` once.
The HA worker may try again later on its own schedule. HA was not tested here.
- **State consistency:** the cleanup changes the state of a network that
another thread is implementing under the lock, and falls back to writing
Shutdown directly to the database.
### versions
=== versions ===
- Observed on: the main nightly
`cloudstack-management-24.0.0-SNAPSHOT.20261027`, which includes the merged
#13907 (f03cffc1; it does not touch this code).
- Code: `AsyncJobManagerImpl.cleanupOrphanedNetworks()`, added by #12409
(215ef98499). It is present on the 4.20 and 4.22 branches and on main.
- Environment:
- KVM, a single host, EL9, MySQL 8.0.
- Isolated networks on DefaultIsolatedNetworkOfferingWithSourceNatService
(not persistent).
- Default `network.gc.interval` and `network.gc.wait` (600/600) and
`job.expire.minutes` (1440).
### The steps to reproduce the bug
=== The steps to reproduce the bug ===
As observed:
1. Take an isolated network on a non-persistent offering, created more than
`job.expire.minutes` ago (default one day).
2. Stop all its VMs and wait until the network GC shuts it down (network in
Shutdown, router stopped). With the defaults this takes about 10-20 minutes.
3. Start a VM on it.
- With a boot group (or any path that calls `advanceStart` once), the
start fails with "Network is not yet fully implemented".
- With the plain startVirtualMachine API, the first attempt fails and the
API's retry usually passes.
4. management-server.log shows `Found orphaned network <id> in Implementing
state without async job` a few seconds after the implementation started.
Quicker variant (not yet run in this exact form):
1. Set `job.expire.minutes` to 5. It is dynamic, but note that it also
shortens job retention on the test system.
2. Create a new isolated network, which stays Allocated until it is first
used.
3. Wait more than 5 minutes, then deploy a VM into it. The cleanup should
cut the first implementation the same way.
Control: a network younger than `job.expire.minutes` is not affected. A
controlled A/B confirmation (a young network against an old one, each case
twice) is running, and its results will follow as a comment.
### What to do about it?
=== What to do about it? ===
Possible fixes (any of these would do):
1. **Measure the time stuck in Implementing, not the age since creation.**
Treat a network as stuck only if it has had no implement activity for X minutes
since it entered Implementing, for example from a timestamp set on that
transition. Use an X tied to how long an implementation can take, not
`job.expire.minutes`.
2. **Recognise the implementer.** Count VmWork jobs (and any other job that
implements networks) as active. Or, before the transition, try the network's
lock without waiting: skip the network if the lock is held, and otherwise
re-read its state under the lock and transition only if it is still
Implementing. Avoid the direct database write to Shutdown while another thread
may own the network.
3. **Tests:**
- Start a VM through `advanceStart` (no retry) on a network older than
the threshold, with the cleanup running. It must succeed, and no orphan may be
logged.
- A network that is truly stuck (lock free, idle for X minutes) is still
cleaned up.
If a fix cannot make 4.20.4, it may be safer to disable this cleanup on the
release branches for now, so that the regression does not ship.
Workarounds until fixed:
- Use a persistent network offering. The network GC never collects those
networks, so they stay Implemented.
- Or make sure networks are Implemented before starts that do not retry, for
example by first starting one VM per network with the plain start API.
cc @Pearl1594: thank you for #12409. We are raising this now only because it
has not shipped yet and there is still time to adjust it before 4.20.4.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
