Apache CloudStack developers,
I’m writing about open issue #13376, where Host-HA can remain stuck in
Fencing when a KVM host is already powered off: Redfish rejects the
repeated GracefulShutdown with HTTP 409, the fence never returns success,
the host never reaches Fenced/Down, and VM-HA therefore never restarts the
affected VMs elsewhere.
The issue’s proposed pre-check for OOBM power status looks like a good fix
for the immediate Redfish case. I think there is also a useful cross-driver
safety invariant worth making explicit:
A fence operation succeeds when the resource is freshly verified to be in
the required isolation/safe target state — not merely when CloudStack
successfully caused a power-state transition.
That distinction covers both sides of the problem:
- If a power-off command fails or returns a non-success code but a fresh
OOBM observation confirms the chassis is Off, the fencing objective has
been achieved and the host can safely transition to Fenced.
- If a power-off command reports success but the resulting host state
cannot be verified, CloudStack should not treat command acceptance alone as
proof that fencing succeeded.
- Unknown, unreachable, stale, or conflicting OOBM state must remain
distinct from a verified Off state; failing open on “unknown” would
undermine the split-brain protection that fencing exists to provide.
A few implementation/test guardrails that may help generalize #13376 beyond
the current Redfish 409 behavior:
1. Separate “transition command result” from “fence-state verification.”
The former is diagnostic evidence; the latter is the HA safety decision.
2. Require a fresh status observation for an already-Off shortcut rather
than trusting a cached power-state value.
3. Make repeated fencing attempts idempotent and convergent: On -> Off
and Off -> Off should both end in the same verified Fenced state when the
BMC remains authoritative/reachable.
4. Preserve the distinction between:
- newly powered Off and verified;
- already Off and verified;
- command failed but subsequent state verified Off;
- command accepted but state still On;
- state unknown/unreachable.
5. Make HA alerts report the actual terminal result. “Fencing performed”
should only mean the safety predicate was verified; failed/unknown attempts
should be distinguishable.
Suggested regression matrix for each OOBM driver used by Host-HA:
- host On, OFF command succeeds, fresh status Off -> fence success;
- host already Off, OFF command returns 409/non-zero, fresh status Off
-> fence success;
- OFF command times out, fresh status Off -> fence success with
diagnostic warning;
- OFF command returns success, fresh status On -> fence failure;
- BMC unreachable / status unknown -> fence failure or unresolved state,
never success;
- cached Off followed by fresh status unknown/On -> do not accept the
stale cache as proof;
- repeated fence attempts against an already-fenced host remain safe and
do not pin the state machine in Fencing.
This would turn the current driver-specific fix into a reusable HA
contract: fencing is about proving the unsafe execution environment can no
longer run the protected workload, not about proving CloudStack was the
actor that transitioned it there.
Issue:
https://github.com/apache/cloudstack/issues/13376
I’m not claiming to have implemented or run a patch; this is a design/test
contribution based on the public issue and the current Host-HA behavior
described there.
Best,
Seth Morrow
Mechanic vs Machine | Independent public-interest review
