With Ivy Bridge-E, Intel introduced their APIC Virtualization extension, APICv
(Rather contradictory, on other sources its called vAPIC). APICv seems to be
built on top of x2APIC, so it requires it to be working on the host to be used.
You can check that in the host by using:
dmesg | grep x2apic
If its returns something like "x2apic enabled", its working. This is something
that users should always check, since it may be possible that the Firmware
tells the OS to NOT use x2APIC.
x2APIC was introduced with Nehalem to supercede the older xAPIC, introduced
with the Pentium 4. Sadly, the initial Intel Tylersburg Chipsets (X58, 5500 and
5520) for LGA 1366 Nehalems had broken Interrupt Remapping support, which
x2APIC seems to rely on. With IR broken, x2APIC didn't worked properly, and
there could be crashes, freezes and such. The solution was to fall back to
older xAPIC.Around that timeframe, instead of blacklisting x2APIC on those
Chipsets OS side with a hotfix/patch so it doesn't defaults to it when
autodetecting platform features, or even better, allow the Firmware to hide the
x2apic CPU Flag which is supposed to inform about x2APIC support, a genius
decided to introduce to the Intel VT-d specifications the "x2APIC Opt Out Bit",
which is a new Bit to tell the OS if the Firmware wants x2APIC to be enabled or
not. The weird part of it is that this Bit resides in the DMAR ACPI Table, so
it seems to require VT-d enabled, since it generates the DMAR ACPI Table. This
sounds like a bad idea, since you STILL have to add OS support for detecting
the Bit instead of merely blacklisting the x2APIC on those Chipsets, along with
a Firmware upgrade for adding it. Not only that, VT-d must be enabled to create
the ACPI DMAR Table, so that Bit is unavailable if the IOMMU (Which also relies
on IR, and I think than that is why the enable_unsafe_interrupts exist in VFIO,
a workaround specifically for that. Or it was another one?) is off. Basically,
the whole thing is a hack and looks like an epic mess.Point is, in my Haswell
Motherboard (A Supermicro X10SAT), a Firmware upgrade introduced an X2APIC Opt
Out Bit option that comes enabled by default. Linux says "DMAR-IR: x2apic is
disabled because BIOS sets x2apic opt out bit" if VT-d is enabled along with
this Bit. So, x2APIC may not work out of the box even in non-broken platforms
because the Firmware may still suggest the OS to not use x2APIC (As a last
resort, it can also be overrided by a Kernel Parameter to force x2APIC if you
fail to figure out what to do Firmware side).
VMs also supports an emulated form of x2APIC, which performs better than the
previous alternative (Very old APIC emulation?? Not sure). Actually, you do not
need host or Processor support for x2APIC, and using it in VMs gives pretty
much free performance - so much that a QEMU dev decided to include the x2APIC
CPU Flag in nearly ALL Processors models (-cpu whatever), so you can use in
QEMU -cpu core2duo and it will by default expose the emulated x2APIC via the
CPU Flag even through that platform didn't had it. You can check in a Linux VM
if x2APIC exist as CPU Flag using...
grep x2apic /proc/cpuinfo
Back to APICv. Some comments I saw in recent months states that the HyperV
hv-vapic extension should NOT be enabled if APICv is available since it has
less overhead, so it performs better. I'm fine with that. The issue is that
there is scarse info regarding what platforms actually supports APICv. Many
people said that Haswell supports it, but I failed to enable it even with
x2APIC working properly (Host side, the emulated x2APIC for VMs is another
thing), and found no solid evidence that Haswell supports APICv at all.
The kvm_intel Kernel Module has the enable_apicv parameter. You can check its
status by using...
cat /sys/module/kvm_intel/parameters/enable_apicv
It should return either a N or Y. In my platform (With a Xeon E3-1245V3
Haswell), by default its N.
The kvm_intel Kernel Module has another rather know parameter which is nested.
You can check its status by using...
cat /sys/module/kvm_intel/parameters/nested
It should return either a N or Y. In my platform, by default its also N.
I can unload kvm_intel, then load it again with appropiated parameters...
modprobe -r kvm_intelmodprobe kvm_intel nested=y
This works as expected, checking nested returns a Y. This is well know, and a
requeriment to enable Nested Virtualization on Intel platforms. But, if in the
same way, I do this...
modprobe -r kvm_intelmodprobe kvm_intel enable_apicv=y
...It has no effect, as checking that parameter still returns a N. Basically,
it seems to point that APICv refuses to enable (Didn't tried as a Kernel
Parameter in Boot Loader, just to make sure it doesn't need to be enabled at
boot time). It may not work probabily because there is no Hardware support for
it...
After a lot of googling, I found absolutely NO info about consumer Haswell
supporting APICv, which is something that other people stated (I think that on
Reddit). What I did found, besides the Ivy Bridge-E announcement, are reviews
mentioning that Haswell-E and Broadwell-E also supports APICv. So, it seems
that APICv is a feature that its exclusive to the enterprise platform (A la
PCIe ACS), since I didn't saw it mentioned for any consumer platform (LGA 1150
Haswell, Broadwell, and LGA 1151 Skylake) at all.
Basically, if you have a Haswell, Broadwell or Skylake consumer platform, you
may want to check for x2APIC status by using...
dmesg | grep x2apic
And if its enabled, you can try checking APICv status by using...
cat /sys/module/kvm_intel/parameters/enable_apicv
Should its possible to determine that consumer platforms doesn't supports APICv
at all, I suppose that you may want to add again hv-vapic for your Windows VMs
if your removed it thinking that your platform supports APICv. I suppose that
the performance scale for VMs goes like this: Emulated x2APIC -> HyperV
hv-vapic (For Windows VMs) -> Intel APICv (On Intel enterprise platforms).
_______________________________________________
vfio-users mailing list
vfio-users@redhat.com
https://www.redhat.com/mailman/listinfo/vfio-users
