From: Sean Christopherson <[email protected]> Sent: Tuesday, April 7, 2026 9:43
AM
>
> +Michael
>
> On Tue, Apr 07, 2026, Vitaly Kuznetsov wrote:
> > Thomas Lefebvre <[email protected]> writes:
> > > Under Hyper-V, raw RDTSC values are not consistent across vCPUs.
> > > The hypervisor corrects them only through the TSC page scale/offset.
> > > If pvclock_update_vm_gtod_copy() runs on CPU 0 and __get_kvmclock()
> > > later runs on CPU 1 where the raw TSC is lower, the unsigned
> > > subtraction wraps.
> > >
> >
> > According to the TLFS, reference TSC page is partition wide:
> >
> > "The hypervisor provides a partition-wide virtual reference TSC page
> > which is overlaid on the partition’s GPA space. A partition’s reference
> > time stamp counter page is accessed through the Reference TSC MSR."
> >
> > so if as you say RAW rdtsc value is inconsistent across vCPUs, I can
> > hardly see how we can use this time source at all, even without
> > KVM. scale/offset are the same for all vCPUs.
> >
> > I think the fix here is to avoid setting up Hyper-V TSC page clocksource
> > in L1. Unfortunately, with unsynchronized TSCs this will leave us the
> > only choice for a sane clocksource: raw HV_X64_MSR_TIME_REF_COUNT MSR
> > reads.
>
> This feels like either a Hyper-V bug or a Linux-as-a-guest bug. For
> "Reference
> Counter"[1]:
>
> The hypervisor maintains a per-partition reference time counter. It has the
> characteristic that successive accesses to it return strictly monotonically
> increasing (time) values as seen by any and all virtual processors of a
> partition. Furthermore, the reference counter is rate constant and
> unaffected
> by processor or bus speed transitions or deep processor power savings
> states. A
> partition’s reference time counter is initialized to zero when the
> partition is
> created. The reference counter for all partitions count at the same rate,
> but
> at any time, their absolute values will typically differ because partitions
> will have different creation times.
>
> The reference counter continues to count up as long as at least one virtual
> processor is not explicitly suspended.
>
>
> And then "Partition Reference Time Enlightenment"[2]:
>
> The partition reference time enlightenment presents a reference time source
> to
> a partition which does not require an intercept into the hypervisor. This
> enlightenment is available only when the underlying platform provides
> support
> of an invariant processor Time Stamp Counter (TSC), or iTSC. In such
> platforms,
> the processor TSC frequency remains constant irrespective of changes in the
> processor’s clock frequency due to the use of power management states such
> as
> ACPI processor performance states, processor idle sleep states (ACPI
> C-states),
> etc.
>
> The partition reference time enlightenment uses a virtual TSC value, an
> offset
> and a multiplier to enable a guest partition to compute the normalized
> reference time since partition creation, in 100nS units. The mechanism also
> allows a guest partition to atomically compute the reference time when the
> guest partition is migrated to a platform with a different TSC rate, and
> provides a fallback mechanism to support migration to platforms without the
> constant rate TSC feature.
>
> My read of "Partition Reference Time Enlightenment" is that it should only be
> advertised if the TSC is synchronized and constant. I can't figure out where
> that feature is actually advertised though, because IIUC it's not the same as
> HV_ACCESS_TSC_INVARIANT, which says that the virtual TSC is guaranteed to be
> invariant even across live migration. And it's not
> HV_MSR_REFERENCE_TSC_AVAILABLE,
> because I'm pretty sure that just says HV_MSR_REFERENCE_TSC is available.
>
> Michael, help?
>
> [1]
> https://learn.microsoft.com/en-us/virtualization/hyper-v-on-windows/tlfs/timers#reference-counter
> [2]
> https://learn.microsoft.com/en-us/virtualization/hyper-v-on-windows/tlfs/timers#partition-reference-time-enlightenment
Yes, TSC page enlightenment is per VM, so it does not compensate for
discrepancies in raw TSC values across physical CPUs. RDTSC in a
Hyper-V VM is executed directly by the hardware (i.e., does not trap to
the hypervisor), so there's no opportunity for the hypervisor to compensate
for discrepancies. The hypervisor is expected to present a VM with TSCs
that are already synchronized. I'll need to double-check, but I don't think
Linux guests on Hyper-V run their own TSC synchronization.
The relevant Hyper-V flags are:
* HV_MSR_TIME_REF_COUNT_AVAILABLE: The synthetic MSR for reading
the partition reference time is available.
* HV_MSR_REFERENCE_TSC_AVAILABLE: The partition reference time
enlightenment (i.e., "the TSC page") is available as a faster way to read
the reference counter.
* HV_ACCESS_TSC_INVARIANT: As Sean said, this says the hardware and
Hyper-V support TSC scaling, so live migration can be done across hosts
without the guest seeing a change in TSC frequency.
Yes, this does feel like an issue where Hyper-V is not presenting the guest
with TSCs that are already synchronized. But I'm not aware of having seen
such a problem before. I'll try to imagine a scenario where a problem like
this could happen via some other path.
@Thomas Lefebvre: Let me double-check a few things via these follow-up
questions/actions:
1. You said the clocksource is hyperv_clocksource_tsc_page. Just to
confirm, that's for the L1 guest, right? Does the output of the "lscpu"
command in the L1 guest show the flags "tsc_reliable" and "constant_tsc"?
I'm assume "no", since if these flags were set, the clocksource (i.e.,
/sys/devices/system/clocksource/clocksource0/current_clocksource)
should be the standard "tsc". I've got a laptop with a i7-13700H processor,
and my L1 VMs show "tsc" as the clocksource, but I haven't been running
KVM with L2 nested VMs.
2. What is the version of Windows/Hyper-V you are running? Get the
output of the "winver.exe" command. It should be something like this:
Windows 11 [as the top banner]
Version 25H2 (OS Build 26200.8037)
3. In the dmesg output of your L1 VM, find the line like this one and reply
with what you have:
Hyper-V: privilege flags low 0xae7f, high 0x3b8030, hints 0x9a4e24, misc
0xe0bed7b2
From there, I can decode the Hyper-V settings and see if anything jumps out
as anomalous.
4. Does the laptop where you are seeing this problem ever hibernate and
then resume? If so, do you recall if the problem occurs after a full reboot but
before it ever does a hibernate/resume cycle?
Michael
