On Wed, Aug 20, 2025 at 7:11 AM Christian Ehrhardt
<christian.ehrha...@canonical.com> wrote:
>
> On Tue, Aug 19, 2025 at 4:51 PM Paolo Bonzini <pbonz...@redhat.com> wrote:
> >
> > On 8/6/25 21:18, Daniel P. Berrangé wrote:
> > > On Wed, Aug 06, 2025 at 07:57:34PM +0200, Christian Ehrhardt wrote:
> > >> On Wed, Aug 6, 2025 at 2:00 PM Daniel P. Berrangé <berra...@redhat.com>
> > >> wrote:
> > >>>
> > >>> On Wed, Aug 06, 2025 at 01:52:17PM +0200, Christian Ehrhardt wrote:
> > >>>> Hi,
> > >>>> I was unsure if this would be better sent to libvirt or qemu - the
> > >>>> issue is somewhere between libvirt modelling CPUs and qemu 10.1
> > >>>> behaving differently. I did not want to double post and gladly most of
> > >>>> the people are on both lists - since the switch in/out of the problem
> > >>>> is qemu 10.0 <-> 10.1 let me start here. I beg your pardon for not yet
> > >>>> having all the answers, I'm sure I could find more with debugging, but
> > >>>> I also wanted to report early for your awareness while we are still in
> > >>>> the RC phase.
> > >>>>
> > >>>>
> > >>>> # Problem
> > >>>>
> > >>>> What I found when testing migrations in Ubuntu with qemu 10.1-rc1 was:
> > >>>> error: operation failed: guest CPU doesn't match specification:
> > >>>> missing features: pdcm
> > >>>>
> > >>>> This is behaving the same with libvirt 11.4 or the more recent 11.6.
> > >>>> But switching back to qemu 10.0 confirmed that this behavior is new
> > >>>> with qemu 10.1-rc.
> > >>>
> > >>>
> > >>>> Without yet having any hard evidence against them I found a few pdcm
> > >>>> related commits between 10.0 and 10.1-rc1:
> > >>>> 7ff24fb65 i386/tdx: Don't mask off CPUID_EXT_PDCM
> > >>>> 00268e000 i386/cpu: Warn about why CPUID_EXT_PDCM is not available
> > >>>> e68ec2980 i386/cpu: Move adjustment of CPUID_EXT_PDCM before
> > >>>> feature_dependencies[] check
> > >>>> 0ba06e46d i386/tdx: Add TDX fixed1 bits to supported CPUIDs
> > >>>>
> > >>>>
> > >>>> # Caveat
> > >>>>
> > >>>> My test environment is in LXD system containers, that gives me issues
> > >>>> in the power management detection
> > >>>> libvirtd[406]: error from service: GDBus.Error:System.Error.EROFS:
> > >>>> Read-only file system
> > >>>> libvirtd[406]: Failed to get host power management capabilities
> > >>>
> > >>> That's harmless.
> > >>
> > >> Yeah, it always was for me - thanks for confirming.
> > >>
> > >>>> And the resulting host-model on a rather old test server will
> > >>>> therefore have:
> > >>>> <cpu mode='custom' match='exact' check='full'>
> > >>>> <model fallback='forbid'>Haswell-noTSX-IBRS</model>
> > >>>> <vendor>Intel</vendor>
> > >>>> <feature policy='require' name='vmx'/>
> > >>>> <feature policy='disable' name='pdcm'/>
> > >>>> ...
> > >>>>
> > >>>> But that was fine in the past, and the behavior started to break
> > >>>> save/restore or migrations just now with the new qemu 10.1-rc.
> > >>>>
> > >>>> # Next steps
> > >>>>
> > >>>> I'm soon overwhelmed by meetings for the rest of the day, but would be
> > >>>> curious if one has a suggestion about what to look at next for
> > >>>> debugging or a theory about what might go wrong. If nothing else comes
> > >>>> up I'll try to set up a bisect run tomorrow.
> > >>>
> > >>> Yeah, git bisect is what I'd start with.
> > >>
> > >> Bisect complete, identified this commit
> > >>
> > >> commit 00268e00027459abede448662f8794d78eb4b0a4
> > >> Author: Xiaoyao Li <xiaoyao...@intel.com>
> > >> Date: Tue Mar 4 00:24:50 2025 -0500
> > >>
> > >> i386/cpu: Warn about why CPUID_EXT_PDCM is not available
> > >>
> > >> When user requests PDCM explicitly via "+pdcm" without PMU enabled,
> > >> emit
> > >> a warning to inform the user.
> > >>
> > >> Signed-off-by: Xiaoyao Li <xiaoyao...@intel.com>
> > >> Reviewed-by: Zhao Liu <zhao1....@intel.com>
> > >> Link:
> > >> https://lore.kernel.org/r/20250304052450.465445-3-xiaoyao...@intel.com
> > >> Signed-off-by: Paolo Bonzini <pbonz...@redhat.com>
> > >>
> > >> target/i386/cpu.c | 3 +++
> > >> 1 file changed, 3 insertions(+)
> > >>
> > >>
> > >>
> > >> Which is odd as it should only add a warning right?
> > >
> > > No, that commit message is misleading.
> > >
> > > IIUC mark_unavailable_features() actively blocks usage of the feature,
> > > so it is a functional change, not merely a emitting warning.
> > >
> > > It makes me wonder if that commit was actually intended to block the
> > > feature or not, vs merely warning ? CC'ing those involved in the
> > > commit.
> > We can revert the commit. I'll send the revert to Stefan and let him
> > decide whether to include it in 10.1-rc4 or delay to 10.2 and 10.1.1.
>
> Thanks Paolo for considering that.
>
> My steps to reproduce seemed really clear and are 100% reproducible
> for me, but no one so far said "yeah they see it too", so I'm getting
> unsure if it was not tried by anyone else or if there is more to it
> than we yet know.
> Further I tested more with the commit reverted, and found that at
> least cross version migrations (9.2 -> 10.1) still have issues that
> seem related - complaining about pdcm as missing feature.
> But that was in a log of a test system that went away and ... you know
> how these things can sometimes be, that new result is not yet very
> reliable.
>
> I intended to check the following matrix more deeply again with and
> without the reverted change and then come back to this thread:
>
> #1 Compare platforms
> - Migrating between non containerized hosts to verify if they are
> affected as well
> - Power management explicitly switched off/on (vs the auto detect of
> host-model) in the guest XML
> #2 Retest the different Use-cases I've seen this pop up
> - 10.1 managed save (broken unless reverting the commit that was identified)
> - 9.2 -> 10.1 migration (seems broken even with the revert)
I'll report back as i can squeeze tests between meetings (there also
is an issue with riscv emulation that I try to identify at the same
time)
For now I managed to eliminate some of my concerns, which was that it
might be my system container based setup.
But I was testing different builds on bare metal x86 this morning and
it looks the same to me.
### ### ###
1:10.0.2+ds-1ubuntu2 (this is what is in the current Ubuntu, until I
started on 10.1)
=> On start maps host-model to:
<model fallback='forbid'>Cascadelake-Server</model>
<vendor>Intel</vendor>
<feature policy='require' name='vmx'/>
<feature policy='require' name='pdcm'/>
...
=> Can do managedsave + start without problems
### ### ###
1:10.1.0~rc2+ds-1ubuntu1 (which has 00268e0 "i386/cpu: Warn about why
CPUID_EXT_PDCM is not available" reverted)
=> On start maps host-model to:
<model fallback='forbid'>Cascadelake-Server</model>
<vendor>Intel</vendor>
<feature policy='require' name='vmx'/>
<feature policy='disable' name='pdcm'/>
^^ this detection is already different due to using
10.1 even with the revert
...
=> Can do managedsave + start without problems
^^ with the revert it works
### ### ###
1:10.1.0~rc3+ds-2ubuntu1~questingppa1 (which is WITHOUT the revert,
more like the normal RC3)
=> On start maps host-model to:
<model fallback='forbid'>Cascadelake-Server</model>
<vendor>Intel</vendor>
<feature policy='require' name='vmx'/>
<feature policy='disable' name='pdcm'/>
^^ same detection as with the revert, but
different than 10.0 formerly did
...
=> Fails to do save and restore just as I had it in my container based tests
$ virsh start testguest
error: Failed to start domain 'testguest'
error: operation failed: guest CPU doesn't match specification:
missing features: pdcm
### ### ###
While I was concerned if my test environment might have been part of
this, it seems it is generally applicable and a problem on bare metal
too.
This is on 6.16.0-13-generic on a Intel Xeon Gold 5222, libvirt 11.6,
only qemu changes between the tests.
With that confirmed, I'd indeed suggest reverting it for now, even
more if someone else could reproduce the same on other platforms and
builds.
On Thu, Aug 7, 2025 at 10:09 AM Xiaoyao Li <xiaoyao...@intel.com> wrote:
>
> So libvirt must first request with "-cpu xxx,pdcm=on" without "pmu=on"
> and gets the result that PDCM is filtered (set in cpu->filtered_features[]).
I wanted to also double down on that former comment, trying to check
logs what they say about pdcm and pmu exactly.
pmu is easy - not mentioned ever in cmdline or configs, but pdcm is
more interesting than I expected.
This is with:
1:10.1.0~rc3+ds-2ubuntu1~questingppa1 (which is WITHOUT the revert,
more like the normal RC3)
The guest gets started at first with this full cpu definition
-cpu
Cascadelake-Server,vmx=on,pdcm=on,hypervisor=on,ss=on,tsc-adjust=on,fdp-excptn-only=on,zero-fcs-fds=on,mpx=on,umip=on,pku=on,md-clear=on,stibp=on,flush-l1d=on,arch-capabilities=on,xsaves=on,ibpb=on,ibrs=on,amd-stibp=on,amd-ssbd=on,rdctl-no=on,ibrs-all=on,skip-l1dfl-vmentry=on,mds-no=on,pschange-mc-no=on,tsx-ctrl=on,sbdr-ssdp-no=on,psdp-no=on,fb-clear=on,rfds-no=on,vmx-ins-outs=on,vmx-true-ctls=on,vmx-store-lma=on,vmx-activity-hlt=on,vmx-activity-wait-sipi=on,vmx-vmwrite-vmexit-fields=on,vmx-apicv-xapic=on,vmx-ept=on,vmx-desc-exit=on,vmx-rdtscp-exit=on,vmx-apicv-x2apic=on,vmx-vpid=on,vmx-wbinvd-exit=on,vmx-unrestricted-guest=on,vmx-apicv-register=on,vmx-apicv-vid=on,vmx-rdrand-exit=on,vmx-invpcid-exit=on,vmx-vmfunc=on,vmx-shadow-vmcs=on,vmx-rdseed-exit=on,vmx-pml=on,vmx-xsaves=on,vmx-tsc-scaling=on,vmx-ept-execonly=on,vmx-page-walk-4=on,vmx-ept-2mb=on,vmx-ept-1gb=on,vmx-invept=on,vmx-eptad=on,vmx-invept-single-context=on,vmx-invept-all-context=on,vmx-invvpid=on,vmx-invvpid-single-addr=on,vmx-invvpid-all-context=on,vmx-invept-single-context-noglobals=on,vmx-intr-exit=on,vmx-nmi-exit=on,vmx-vnmi=on,vmx-preemption-timer=on,vmx-posted-intr=on,vmx-vintr-pending=on,vmx-tsc-offset=on,vmx-hlt-exit=on,vmx-invlpg-exit=on,vmx-mwait-exit=on,vmx-rdpmc-exit=on,vmx-rdtsc-exit=on,vmx-cr3-load-noexit=on,vmx-cr3-store-noexit=on,vmx-cr8-load-exit=on,vmx-cr8-store-exit=on,vmx-flexpriority=on,vmx-vnmi-pending=on,vmx-movdr-exit=on,vmx-io-exit=on,vmx-io-bitmap=on,vmx-mtf=on,vmx-msr-bitmap=on,vmx-monitor-exit=on,vmx-pause-exit=on,vmx-secondary-ctls=on,vmx-exit-nosave-debugctl=on,vmx-exit-load-perf-global-ctrl=on,vmx-exit-ack-intr=on,vmx-exit-save-pat=on,vmx-exit-load-pat=on,vmx-exit-save-efer=on,vmx-exit-load-efer=on,vmx-exit-save-preemption-timer=on,vmx-exit-clear-bndcfgs=on,vmx-entry-noload-debugctl=on,vmx-entry-ia32e-mode=on,vmx-entry-load-perf-global-ctrl=on,vmx-entry-load-pat=on,vmx-entry-load-efer=on,vmx-entry-load-bndcfgs=on,vmx-eptp-switching=on,hle=off,rtm=off
which has "pdcm=on" but does not mention pmu either way.
But once qemu starts we see it emit
2025-08-20T08:53:00.514650Z qemu-system-x86_64: warning: This feature
is not available due to PMU being disabled: CPUID[eax=01h].ECX.pdcm
[bit 15]
When I do "virsh start" after "managedsave" I see this:
-cpu
Cascadelake-Server,vmx=on,pdcm=off,hypervisor=on,ss=on,tsc-adjust=on,fdp-excptn-only=on,zero-fcs-fds=on,mpx=on,umip=on,pku=on,md-clear=on,stibp=on,flush-l1d=on,arch-capabilities=on,xsaves=on,ibpb=on,ibrs=on,amd-stibp=on,amd-ssbd=on,rdctl-no=on,ibrs-all=on,skip-l1dfl-vmentry=on,mds-no=on,pschange-mc-no=on,tsx-ctrl=on,sbdr-ssdp-no=on,psdp-no=on,fb-clear=on,rfds-no=on,vmx-ins-outs=on,vmx-true-ctls=on,vmx-store-lma=on,vmx-activity-hlt=on,vmx-activity-wait-sipi=on,vmx-vmwrite-vmexit-fields=on,vmx-apicv-xapic=on,vmx-ept=on,vmx-desc-exit=on,vmx-rdtscp-exit=on,vmx-apicv-x2apic=on,vmx-vpid=on,vmx-wbinvd-exit=on,vmx-unrestricted-guest=on,vmx-apicv-register=on,vmx-apicv-vid=on,vmx-rdrand-exit=on,vmx-invpcid-exit=on,vmx-vmfunc=on,vmx-shadow-vmcs=on,vmx-rdseed-exit=on,vmx-pml=on,vmx-xsaves=on,vmx-tsc-scaling=on,vmx-ept-execonly=on,vmx-page-walk-4=on,vmx-ept-2mb=on,vmx-ept-1gb=on,vmx-invept=on,vmx-eptad=on,vmx-invept-single-context=on,vmx-invept-all-context=on,vmx-invvpid=on,vmx-invvpid-single-addr=on,vmx-invvpid-all-context=on,vmx-invept-single-context-noglobals=on,vmx-intr-exit=on,vmx-nmi-exit=on,vmx-vnmi=on,vmx-preemption-timer=on,vmx-posted-intr=on,vmx-vintr-pending=on,vmx-tsc-offset=on,vmx-hlt-exit=on,vmx-invlpg-exit=on,vmx-mwait-exit=on,vmx-rdpmc-exit=on,vmx-rdtsc-exit=on,vmx-cr3-load-noexit=on,vmx-cr3-store-noexit=on,vmx-cr8-load-exit=on,vmx-cr8-store-exit=on,vmx-flexpriority=on,vmx-vnmi-pending=on,vmx-movdr-exit=on,vmx-io-exit=on,vmx-io-bitmap=on,vmx-mtf=on,vmx-msr-bitmap=on,vmx-monitor-exit=on,vmx-pause-exit=on,vmx-secondary-ctls=on,vmx-exit-nosave-debugctl=on,vmx-exit-load-perf-global-ctrl=on,vmx-exit-ack-intr=on,vmx-exit-save-pat=on,vmx-exit-load-pat=on,vmx-exit-save-efer=on,vmx-exit-load-efer=on,vmx-exit-save-preemption-timer=on,vmx-exit-clear-bndcfgs=on,vmx-entry-noload-debugctl=on,vmx-entry-ia32e-mode=on,vmx-entry-load-perf-global-ctrl=on,vmx-entry-load-pat=on,vmx-entry-load-efer=on,vmx-entry-load-bndcfgs=on,vmx-eptp-switching=on,hle=off,rtm=off
I still get this warning
2025-08-20T08:54:54.477469Z qemu-system-x86_64: warning: This feature
is not available due to PMU being disabled: CPUID[eax=01h].ECX.pdcm
[bit 15]
But more interestingly - in this second start, the one with a managed
save present, we see libvirt call the guest with
pdcm=off
So for a yet unknown reason this second start is setting pdcm=off and
therefore all that follows is expected.
And yep, then we will get
error: operation failed: guest CPU doesn't match specification:
missing features: pdcm
So we are partially back to my original suspicion that is is somewhere
in between qemu and libvirt, it is neither alone but their interaction
that is breaking this AFAICS.
> The hope was that these will help to further identify what is going
> on, but despite the urgency of the release being imminent I have not
> yet managed to find the time in the last two days :-/
>
> > Sorry for the delay in answering (and thanks Daniel for bringing this to
> > my attention).
> >
> > Thanks,
> >
> > Paolo
> >
>
>
> --
> Christian Ehrhardt
> Director of Engineering, Ubuntu Server
> Canonical Ltd
--
Christian Ehrhardt
Director of Engineering, Ubuntu Server
Canonical Ltd
