nikolauseppinger opened a new pull request, #13589:
URL: https://github.com/apache/cloudstack/pull/13589
### Description
When a KVM host suffers a complete power failure, its BMC (IPMI/iDRAC/iLO)
usually loses power together with the host. The Host HA framework's fence
operation (`KVMHAProvider.fence()`) relies solely on out-of-band management
(`PowerOperation.OFF`), so fencing can never succeed in this scenario:
- The FSM has no exit from `Fencing` except `Fenced` or `Disabled`, so the
host retries fencing forever (`RetryFencing`).
- `FenceTask.processResult()` only calls
`oldHighAvailabilityManager.scheduleRestartForVmsOnHost()` on a successful
fence, so HA-enabled VMs are never restarted.
- `HAManagerImpl.isVMAliveOnHost()` returns true for every state except
`Fenced`, which also blocks the legacy VM HA investigation path.
The result: on a total power failure — arguably the most common
total-failure scenario HA exists for — all HA-enabled VMs stay down
indefinitely until an operator intervenes (e.g. via `declareHostAsDegraded`).
Related reports: #12921, #12185.
At the point `fence()` is invoked, CloudStack has already gathered strong
evidence that the host is dead:
1. Health checks failed (agent gone),
2. The configured number of activity checks found no VM disk activity on any
pool (`CheckVMActivityOnStoragePoolCommand`),
3. Neighbouring hosts positively report the host's storage heartbeat as
expired — the same evidence the legacy `KVMFencer` has always accepted as
sufficient proof of fencing.
### Change
This PR adds an **opt-in** fallback, disabled by default:
- New cluster-scope setting `kvm.ha.fence.on.storage.heartbeat`
(default `false`).
- When the OOBM fence operation fails (unsuccessful response, OOBM disabled,
or an exception such as an unreachable BMC), the host is considered fenced
**only** if the neighbouring hosts positively report its storage heartbeat
as expired (`Status.Down` from
`KVMHostActivityChecker.getHostAgentStatus()`,
which requires neighbour confirmation via `CheckOnHostCommand`).
- The host is **never** considered fenced when any check still sees it alive,
when no neighbour can confirm (`Disconnected`/`Unknown`), or when the
heartbeat check itself fails — absence of evidence never counts as evidence
of death.
- With the setting disabled (default), behaviour is completely unchanged.
The heartbeat semantics follow the existing checks, including
`kvm.ha.fence.host.if.heartbeat.fails.on.storage`.
### Safety considerations
Fencing exists to guarantee the host can no longer write to shared storage
before VMs are restarted elsewhere. The fallback only fires after three
independent confirmations (failed health checks, no VM disk activity over the
whole activity-check window, expired storage heartbeat confirmed by
neighbours), which is strictly more evidence than the legacy `KVMFencer`
requires for the same decision. A transient BMC outage on a live host does
not trigger it: a live host keeps writing its heartbeat, so neighbours will
not report `Down`.
### Types of changes
- [x] Enhancement (improves an existing feature and functionality)
- [x] Bug fix (non-breaking change which fixes an issue)
### Bug Severity
- [x] Major
### How Has This Been Tested?
- 6 new unit tests in `KVMHostHATest` covering all fence paths (OOBM success,
OOBM failure with the setting disabled/enabled, heartbeat dead/alive, OOBM
exception with and without fallback); 10/10 tests pass.
- `mvn -pl plugins/hypervisors/kvm -am test` builds green including
checkstyle.
- Scenario observed on 4.22.1.0 (KVM on RHEL 9.8, SharedMountPoint primary
storage, iDRAC OOBM): host power loss took down the iDRAC, host remained in
Fencing with all HA VMs down; neighbour heartbeat checks had reported the
host dead 62 seconds after the failure.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
