andrijapanicsb opened a new issue, #13824:
URL: https://github.com/apache/cloudstack/issues/13824
## Summary
Apache CloudStack 4.22 gives all Windows guests a `localtime` clock and
`hypervclock`, but it emits a `<hyperv>` feature block only when the guest OS
label contains `Windows PV`. Even in that case, the block is limited to:
```xml
<hyperv>
<relaxed state='on'/>
<vapic state='on'/>
<spinlocks state='on' retries='8096'/>
</hyperv>
```
This is a historical, very small subset of the Hyper-V enlightenments
supported
by every KVM host family currently listed in the CloudStack 4.22
compatibility
matrix. The generic `Windows PV` label also does not describe a Windows
version
and does not, by itself, select VirtIO storage or networking.
CloudStack should introduce an explicit, versioned Windows-on-KVM
enlightenment
profile instead of relying on the substring `Windows PV` as the only switch.
## Current behavior
In `LibvirtComputingResource.enlightenWindowsVm()` CloudStack checks only:
```java
if (vmTO.getOs().contains("Windows PV")) {
// relaxed, vapic and spinlocks only
}
```
`LibvirtVMDef.HyperVEnlightenmentFeatureDef` can currently serialize only
those
three features. Separately, every OS label beginning with `Windows` receives
a
`hypervclock` timer.
Consequences:
- Windows Server 2012 through Windows Server 2025 and current Windows desktop
guests do not receive a `<hyperv>` block when their real version-specific
OS
type is selected.
- Selecting `Windows PV` loses version information while enabling only three
old enlightenments.
- The effective behavior is hard-coded rather than represented as a
documented
compatibility profile.
- CloudStack cannot safely add newer features according to host capabilities
or
preserve a stable feature set for live migration within a cluster.
## Why this can be modernized without requiring VirtIO drivers
Hyper-V enlightenments are CPU/timer/interrupt interfaces exposed by KVM and
QEMU. They are independent of the guest's disk and network device models. A
Windows VM can use these features while continuing to boot from IDE or SATA
and
using an emulated E1000 NIC.
This issue therefore must not implicitly change disk controllers, NIC models,
firmware, or machine type. Those changes have separate guest-driver and boot
compatibility risks.
## CloudStack 4.22 KVM compatibility floor
The 4.22 compatibility matrix lists the following KVM host OS families:
- Ubuntu 22.04 and 24.04 LTS
- Oracle Linux, Rocky Linux, AlmaLinux and RHEL 8, 9 and 10
- openSUSE Leap 15 and SLES 15
CloudStack documents major OS families and does not pin a minimum
service-pack,
point-release, kernel, QEMU package build, or libvirt package build.
Therefore
the table below is a conservative engineering floor based on the oldest
branch
named by CloudStack, not a claim that every EOL initial point-release is
tested
or should be deployed.
| CloudStack 4.22 host family | Conservative oldest branch reference | QEMU
family | libvirt family | Linux kernel family | Qualification |
|---|---:|---:|---:|---:|---|
| Ubuntu | 22.04 LTS | 6.2 | 8.0 | 5.15 | Explicitly listed by CloudStack |
| RHEL / Oracle Linux | 8.0-era baseline | 2.12 | 4.5 | 4.18 | RHEL is
listed as expected to work; current patched minor releases are required in
practice |
| Rocky / AlmaLinux | EL8 baseline | no lower than the EL8 floor | no lower
than the EL8 floor | 4.18 | These distributions first shipped after EL8.0; use
the lower EL8 family floor for design |
| openSUSE Leap / SLES | 15.0-era baseline | 2.11 | 4.0 | 4.12 | CloudStack
says Leap 15.6 is the tested Leap release; the older numbers are only a
conservative branch floor |
The lowest conservative host in that matrix is still substantially newer than
the minimum required for the proposed baseline below.
## Proposed implementation
### 1. Add explicit profiles and version-based defaults
Add a saved Windows virtualization profile with four values:
- `disabled`: no `<hyperv>` block; retain the existing Windows clock
handling.
- `legacy`: exactly the current `Windows PV` feature set: `relaxed`, `vapic`,
and `spinlocks retries='8096'`.
- `modern-safe`: the CPU-vendor-neutral, cross-distro baseline described
below.
- `auto`: `modern-safe` plus optional features supported by every eligible
host
in the selected KVM cluster.
Use the following defaults for newly deployed VMs:
| Selected guest OS | Default profile | Reason |
|---|---|---|
| Windows 8/8.1/10/11 | `modern-safe` | Version-specific modern Windows type
|
| Windows Server 2012/2012 R2/2016/2019/2022/2025 and newer | `modern-safe`
| Version-specific modern Windows Server type |
| Windows 7 and Windows Server 2008/2008 R2 | `legacy` | Preserve the small
feature set originally introduced for Windows Server 2008 and later |
| Windows Vista/XP, Windows Server 2003 and older | `disabled` | Do not
advertise untested features to legacy guests |
| Versionless `Windows PV` | `legacy` | Preserve current CloudStack behavior
|
| Unknown or unclassified Windows type | `disabled` | Prefer predictable
compatibility when the Windows generation is not known |
The operator can override the proposed value before deployment. Existing VMs
must keep their current generated XML when no saved profile exists: existing
`Windows PV` VMs continue to behave as `legacy`, while other existing Windows
VMs continue without a `<hyperv>` block. An upgrade alone must not alter a
VM's
virtual CPU definition.
The selected and effective values must be visible in debug logs and in the
generated domain XML.
### 2. Expose a clear Windows-only UI and API setting
Show a field labelled **Windows virtualization optimizations** when the
selected
hypervisor is KVM and the selected guest OS is Windows. Hide it for
non-Windows
guests and non-KVM hypervisors.
The field is automatically populated according to the table above and offers:
- **Modern (recommended)** ??? portable modern optimizations supported
across the
CloudStack 4.22 KVM host baseline.
- **Automatic** ??? Modern plus additional optimizations supported by every
host
in the target KVM cluster.
- **Legacy compatibility** ??? current `Windows PV` behavior.
- **Disabled** ??? no Hyper-V feature block.
The management server must be the authoritative source for the supported
profile values and their stable identifiers. The UI must populate the
drop-down
from backend-provided values, following the same general pattern used for
other
VM settings such as keyboard type and root disk controller. Do not maintain
an
independent frontend-only list that can drift from backend validation.
The default UI should not expose individual QEMU feature names or the raw
spinlock retry number. A tooltip or help panel must explain that the setting
controls Hyper-V-compatible CPU, timer, and interrupt optimizations exposed
by
KVM; it does not install or require VirtIO storage or network drivers. Each
option must have a short description and a link to administrator
documentation.
Store the template's proposed/default value in template details. At
deployment,
copy the selected value to VM details. From that point, the VM value is the
authoritative setting used for every start, restart, and reschedule.
Changing it
after deployment must require the VM to be stopped. This prevents a later
CloudStack upgrade, template edit, or move to another host from silently
changing the virtual CPU definition.
Expose the same drop-down in VM Settings. When the VM is stopped, an
operator can
select another backend-provided profile and save it to VM details. When the
VM
is running, show the effective value as read-only and explain that the VM
must
be stopped before it can be changed. The newly selected profile is applied
when
CloudStack next generates the domain XML and starts the VM.
The API must expose the supported values, accept the selected optional value
at
deployment, and provide an update operation for a stopped VM so API and UI
behavior remain identical. Backend validation must reject unknown values and
changes attempted while the VM is running. Existing callers that omit the
value
retain the compatibility behavior described above.
### 3. Implement a universal `modern-safe` baseline
Use an explicit/custom feature list rather than libvirt Hyper-V passthrough:
```xml
<hyperv mode='custom'>
<relaxed state='on'/>
<vapic state='on'/>
<spinlocks state='on' retries='4095'/>
<vpindex state='on'/>
<runtime state='on'/>
<synic state='on'/>
<stimer state='on'/>
</hyperv>
```
The highest host requirement in this baseline is:
- **QEMU 2.6 or newer**
- **libvirt 1.3.3 or newer**
This is below the conservative QEMU/libvirt floor of every KVM distribution
family listed by CloudStack 4.22. It is independent of Intel versus AMD host
CPU
and does not require VirtIO drivers in the Windows guest.
`spinlocks retries` is a retry count, not a time value or network port. When
one
virtual CPU is waiting for a lock potentially held by another virtual CPU,
the
guest retries that many times before sending the advisory
`HvCallNotifyLongSpinWait` hypercall. The notification helps the hypervisor
make
a better scheduling decision, especially on an overcommitted host where the
lock-holding virtual CPU may not currently be running.
CloudStack introduced the decimal value `8096` in the 2018 enlightenment
change,
but its commit, pull request, tests, and review discussion contain no
explanation
for that value. There is no technical relationship between this retry count
and
CloudStack's optional integration API port 8096; the identical decimal number
appears to be coincidental. Preserve `8096` only in the `legacy` profile so
that
existing behavior remains reproducible. Use `4095` (`0xfff`) for
`modern-safe`,
which is the current QEMU recommendation for overcommitted hosts. QEMU
documents
`0xffffffff` as the effective "never notify" choice when this scheduling
hint is
not desired.
### 4. Add newer features through the `auto` profile
Do not add the following to the unconditional baseline:
| Optional feature | Minimum libvirt | Minimum QEMU | Proposed policy |
|---|---:|---:|---|
| `frequencies` | 4.7.0 | 2.12 | `auto` tier only; required dependency for
re-enlightenment |
| `reenlightenment` | 4.7.0 | 3.0 | `auto` tier; validate migration behavior
and CPU frequency exposure |
| `tlbflush` | 4.7.0 | 3.0 | `auto` tier; requires `vpindex` |
| `ipi` | 4.10.0 | 3.1 | `auto` tier; requires `vpindex` |
| `stimer direct` | 5.7.0 | 4.1 | Newer-host tier only |
| `evmcs` | 4.10.0 | 3.1 | Explicit opt-in only for nested Hyper-V on
compatible Intel hosts; not a universal Intel/AMD feature |
| `avic` | 8.10.0 | 6.2 | Do not enable generically;
hardware/topology-specific |
| `tlbflush direct/extended` | 11.0.0 | 7.1 | Future tier only; outside the
oldest CloudStack 4.22 host-family floor |
Feature dependencies must be encoded and tested, for example `stimer` depends
on `synic`, `synic` depends on `vpindex`, and `reenlightenment` depends on
`frequencies`.
The `modern-safe` profile must not perform Intel/AMD or per-host capability
checks. It is the portable baseline.
The `auto` profile is part of this proposal and may be delivered as a
follow-up
PR after the portable baseline and UI/API setting. It must use the
intersection
of libvirt/QEMU capabilities across every host eligible to run the VM. Store
both
the requested value (`auto`) and the resolved feature list on the VM so that
its
virtual CPU definition remains stable across restart and migration. If a
target
host cannot provide that saved feature list, CloudStack must reject that
target
rather than silently reducing the list.
`mode='passthrough'` must not be used because it can expose a different
feature
set on different hosts and make live migration unsafe. Optional features with
additional hardware requirements, including `evmcs`, `avic`, and
`reenlightenment`, must not be enabled merely because the installed QEMU
version
recognizes their XML.
### 5. Extend the domain XML model and Windows classification
- Extend `HyperVEnlightenmentFeatureDef` to serialize the baseline and
optional
feature set, including nested attributes only where required.
- Replace the single `contains("Windows PV")` decision with guest-family and
guest-version-aware selection.
- Preserve `hypervclock` and existing local-time behavior.
- Persist the chosen/effective profile so rescheduling and restart generate
the
same virtual CPU definition.
- Reject an unsupported explicitly requested profile with a clear, actionable
error; never silently reduce the requested feature set or emit XML the
destination cluster cannot support.
## Compatibility and rollout requirements
- No change to an existing VM merely because CloudStack was upgraded.
- No automatic IDE/SATA-to-VirtIO, E1000-to-VirtIO, BIOS-to-UEFI, or
i440FX-to-Q35 conversion as part of this issue.
- No dependence on VirtIO drivers.
- The baseline must work on both Intel and AMD KVM hosts.
- The effective feature set must be stable across all hosts eligible for live
migration of the VM.
- Apply the version-based defaults only to newly deployed VMs. Preserve the
generated XML of existing VMs unless the operator explicitly changes it.
- Document that changing a VM's enlightenment profile changes its virtual CPU
definition and must be performed while stopped.
## Acceptance criteria
1. Unit tests verify exact XML for `disabled`, `legacy`, `modern-safe`, and
each
supported `auto` capability tier.
2. Unit tests verify guest classification for version-specific Windows types,
`Windows PV`, non-Windows guests, and null/unknown OS mappings.
3. The `modern-safe` XML can be defined and started on the oldest supported
CloudStack CI images or equivalent test hosts for the Ubuntu 22.04, EL8,
and SUSE 15 families.
4. Boot, reboot, clean shutdown, timekeeping, and multi-vCPU stress are
tested
with Windows Server 2012 R2, 2016, 2019, 2022, and 2025.
5. At least one test uses only emulated storage/network devices to prove that
VirtIO drivers are not a prerequisite.
6. Intel and AMD hosts are covered.
7. Cold and live migration are tested within a homogeneous cluster and across
hosts with different QEMU/libvirt patch levels but the same negotiated
effective profile.
8. An existing VM without profile metadata retains the pre-change XML.
9. Unsupported optional features produce a clear validation error; CloudStack
does not silently reduce a saved or explicitly selected feature set.
10. Administrator documentation describes profiles, host-version boundaries,
guest-version mapping, override behavior, and rollback.
11. UI tests verify that the field is shown only for Windows on KVM,
receives the
correct version-based default, can be overridden, and includes
explanatory
help text.
12. Tests preserve `8096` in `legacy` and emit `4095` in `modern-safe`.
13. The VM Settings drop-down is populated from backend-provided values, is
editable for a stopped VM, is read-only for a running VM, and persists
the
selected value in VM details.
14. API tests verify supported-value discovery, deployment-time selection,
stopped-VM updates, rejection of unknown values, and rejection of changes
while a VM is running.
## Out of scope
- Selecting VirtIO disk or NIC models
- Installing or validating Windows VirtIO drivers
- Automatically converting existing disk controllers or firmware
- Changing BIOS/i440FX versus UEFI/Q35 defaults
- Video-device modernization
- Enabling nested Hyper-V or `evmcs` by default
## References
- [CloudStack 4.22 compatibility
matrix](https://docs.cloudstack.apache.org/en/4.22.0.0/releasenotes/compat.html)
- [Current CloudStack KVM Windows feature and clock
generation](https://github.com/apache/cloudstack/blob/4.22.0.0/plugins/hypervisors/kvm/src/main/java/com/cloud/hypervisor/kvm/resource/LibvirtComputingResource.java)
- [Current CloudStack Hyper-V feature XML
model](https://github.com/apache/cloudstack/blob/4.22.0.0/plugins/hypervisors/kvm/src/main/java/com/cloud/hypervisor/kvm/resource/LibvirtVMDef.java)
- [Original CloudStack Windows enlightenment
change](https://github.com/apache/cloudstack/commit/c4b621a418fe74a56e68140f1aa35c70562d7248)
- [libvirt domain XML: Hyper-V features and minimum
versions](https://libvirt.org/formatdomain.html#hypervisor-features)
- [QEMU Hyper-V enlightenment behavior and
recommendations](https://www.qemu.org/docs/master/system/i386/hyperv.html)
- [Microsoft Hyper-V guest spinlock
behavior](https://learn.microsoft.com/en-us/virtualization/hyper-v-on-windows/tlfs/vp-properties)
- [Microsoft
`HvCallNotifyLongSpinWait`](https://learn.microsoft.com/en-us/virtualization/hyper-v-on-windows/tlfs/hypercalls/hvcallnotifylongspinwait)
- [Ubuntu 22.04 QEMU
package](https://packages.ubuntu.com/jammy/qemu-system-x86)
- [Ubuntu 22.04 libvirt
package](https://packages.ubuntu.com/jammy/admin/libvirt-daemon-system)
## Decisions proposed by this issue
- Default new, version-specific Windows 8 / Server 2012 and newer VMs to
`modern-safe`; default older and unknown types according to the explicit
table
above.
- Show a documented, operator-overridable Windows-only setting in both the UI
and API.
- Populate the UI drop-down from the backend's authoritative profile list and
allow the saved value to be changed from VM Settings while the VM is
stopped.
- Save the template default and copy the selected value to the VM so
subsequent
starts use the same virtual CPU definition.
- Ship the CPU-vendor-neutral `modern-safe` baseline without host capability
negotiation.
- Include `auto` in the overall issue scope, but allow its cluster capability
implementation to be a follow-up PR.
- Preserve retries `8096` only for exact legacy compatibility and use QEMU's
documented overcommit recommendation, `4095`, for `modern-safe`.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
