andrijapanicsb commented on issue #14522:
URL: https://github.com/apache/cloudstack/issues/14522#issuecomment-6493745966
A/B confirmation, as promised in the issue. Same build (main nightly
`cloudstack-management-24.0.0-SNAPSHOT.20261027`), KVM, a single host, isolated
networks on DefaultIsolatedNetworkOfferingWithSourceNatService (not
persistent). The global settings were unchanged: `network.gc.interval` /
`network.gc.wait` 600/600 and `job.expire.minutes` 1440, read back after the
test.
**Method.** Each case starts one VM on one network that needs implementing.
The cases ran one at a time, about 30 s apart, with young and old interleaved.
Before the test, the network GC had shut every network in A, B and P down
(state Allocated, router stopped).
- **A (control):** networks 211 and 212, created 2026-10-29 04:50 UTC (22
and 26 minutes old at the test). Each was implemented once by a first VM start,
then the VM was stopped and the GC shut the network down. A one-VM boot group
starts the VM (`advanceStart`, no retry).
- **B:** networks 206 and 207, created 2026-10-27 (about 38.6 h old). The
same one-VM boot group start.
- **P:** networks 208 and 209 (old, shut down by the GC). A plain
`startVirtualMachine` (the API path, which retries once).
- **Q (age by backdating):** networks 213 and 214, new and never implemented
(Allocated). The age condition comes from backdating `networks.created` on
these two test networks only, while they were Allocated:
`UPDATE networks SET created = created - INTERVAL 2 DAY WHERE id IN (213,
214);`
- Q1: `deployVirtualMachine` into network 213.
- Q2: a one-VM boot group on network 214.
**Results:**
| Case | Network | Age at test | Path | Result | Cleanup during the
implementation |
|---|---|---|---|---|---|
| B1 | 206 | 2318 min | boot group | **failed** after 44 s | orphaned 3.9 s
after the implementation started |
| A1 | 211 | 22 min | boot group | passed, 75 s | saw it Implementing 4
times, no orphan |
| B2 | 207 | 2322 min | boot group | **failed** after 45 s | orphaned 1.9 s
after the implementation started |
| A2 | 212 | 26 min | boot group | passed, 76 s | saw it Implementing 4
times, no orphan |
| P1 | 208 | 2326 min | plain start | passed, 55 s; **first attempt
failed**, retry passed | orphaned 5.9 s after the implementation started |
| P2 | 209 | 2327 min | plain start | passed, 53 s; **first attempt
failed**, retry passed | orphaned 9.9 s after the implementation started |
| Q1 | 213 | 2880 min (backdated) | deploy | passed, 51 s; **first attempt
failed**, retry passed | orphaned 9.9 s after the implementation started |
| Q2 | 214 | 2881 min (backdated) | boot group | **failed** after 43 s |
orphaned 7.9 s after the implementation started |
- In every old or backdated case, 38-41 s passed from the start of the
implementation to the router being up, and a 10-second cleanup run landed
inside that window.
- The young networks took about as long (37-38 s) and were seen by 4 cleanup
runs each, which left them alone.
- After the failed boot-group starts, the networks stayed in Allocated.
**Log lines (management-server.log, 2026-10-29 UTC; times exact, lines
shortened, network ids only):**
```
B1 05:11:07.717 lock acquired for network 206 (implement)
05:11:11.647 Found 1 networks in Implementing state, checking for
orphaned networks
05:11:11.648 Found orphaned network 206 in Implementing state without
async job. ... age: 2318 minutes, expiration threshold: 1440 minutes.
Transitioning to Shutdown state.
05:11:46.994 router start completed
05:11:48.859 ResourceUnavailableException: Resource [Network:206] is
unreachable: Network is not yet fully implemented
05:11:49.266 boot group start halted (tier 0: failed to start a VM)
A1 05:12:43.711 lock acquired for network 211 (implement)
05:12:51.641 / 05:13:01.648 / 05:13:11.639 / 05:13:21.644 Found 1
networks in Implementing state (no orphan logged)
05:13:21.052 router start completed
05:13:25.797 VM start completed
B2 05:14:49.751 lock acquired for network 207 (implement)
05:14:51.645 Found orphaned network 207 ... age: 2322 minutes ...
Transitioning to Shutdown state.
05:15:30.810 router start completed
05:15:32.480 Resource [Network:207] is unreachable: Network is not yet
fully implemented
A2 05:16:27.717 lock acquired for network 212 (implement)
05:16:31.641 / 05:16:41.639 / 05:16:51.649 / 05:17:01.642 Found 1
networks in Implementing state (no orphan logged)
05:17:05.217 router start completed
05:17:10.593 VM start completed
P1 05:18:35.740 lock acquired for network 208 (implement), first attempt
05:18:41.655 Found orphaned network 208 ... age: 2326 minutes ...
Transitioning to Shutdown state.
05:19:14.872 router start completed
05:19:17.107 Resource [Network:208] is unreachable: Network is not yet
fully implemented
05:19:19.751 lock acquired for network 208 (implement), retry; router
already running
05:19:26.278 VM start completed
P2 05:20:21.732 lock acquired for network 209 (implement), first attempt
05:20:31.647 Found orphaned network 209 ... age: 2327 minutes ...
05:21:01.127 router start completed
05:21:02.889 Resource [Network:209] is unreachable: Network is not yet
fully implemented
05:21:03.748 lock acquired for network 209 (implement), retry
05:21:10.961 VM start completed
Q1 05:21:41.743 lock acquired for network 213 (implement), first attempt
05:21:51.642 Found orphaned network 213 ... Network created:
2026-10-27T05:21:34 (backdated), age: 2880 minutes ...
05:22:19.817 router start completed
05:22:20.923 Resource [Network:213] is unreachable: Network is not yet
fully implemented
05:22:21.756 lock acquired for network 213 (implement), retry
05:22:28.415 VM start completed
Q2 05:23:23.775 lock acquired for network 214 (implement)
05:23:31.646 Found orphaned network 214 ... age: 2881 minutes ...
05:24:01.758 router start completed
05:24:02.807 Resource [Network:214] is unreachable: Network is not yet
fully implemented
05:24:03.144 boot group start halted (tier 0: failed to start a VM)
```
**Conclusion.** The network's age since creation alone decides the outcome.
- Every start that implemented a network older than `job.expire.minutes` had
its network moved to Shutdown mid-implementation: 6 of 6, including 2 that were
only backdated.
- Every start on a young network passed: 2 of 2.
- Paths without a retry (boot groups) failed: 3 of 3.
- The API path passed only because of its one retry: 3 of 3, each with a
failed first attempt.
**The retry is not a safe net either.** Shortly after the A/B, a plain
`deployVirtualMachine` into old network 209 (shut down by the GC) failed
outright. The cleanup caught both attempts:
```
06:26:09.819 lock acquired for network 209 (implement), first attempt
06:26:13.743 Found orphaned network 209 ... age: 2393 minutes ...
Transitioning to Shutdown state.
06:26:47.197 router start completed
06:26:49.501 Resource [Network:209] is unreachable: Network is not yet
fully implemented
06:26:51.917 lock acquired for network 209 (implement), retry; router
already running
06:26:53.745 Found orphaned network 209 ... age: 2394 minutes ...
Transitioning to Shutdown state.
06:26:53.871 Resource [Network:209] is unreachable: Network is not yet
fully implemented
06:26:54.337 deploy job FAILED (530); the VM ends in Error, with the alert
"Failed to deploy Instance"
```
The retry's implementation took about 2 s, and a 10-second cleanup run
landed inside it. A new deploy about two minutes later passed.
**For maintainers reproducing it:** backdating a test network's `created`,
as above, reproduces it immediately. Lowering `job.expire.minutes` also
reproduces it, but it deletes completed job history; backdating a test
network's `created` has no side effects.
**Updated counts from the soak described in the issue:** the issue was
written at 7 cleanup runs and 17 failed boot group starts. By the end of the
soak (95 cycles, ending 2026-10-29 04:49 UTC), it was 10 cleanup runs and 26
failed boot group starts. Leaving aside readiness-rule timing (a separate
boot-group topic), all but one of the soak's failed group starts came from this
cleanup.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
