Consider the following circumstances:
- An x86-64 multicore system is running with all cores set for long mode
(EFER.LME and EFER.LMA set)
- The OS decides to re-launch one of the AP CPUs using an INIT IPI
According to the Intel architecture manual, an INIT IPI should reset the CPU
state (with a few small exceptions):
[...]
10.4.7.3 Local APIC State After an INIT Reset ("Wait-for-SIPI" State)
An INIT reset of the processor can be initiated in either of two ways:
· By asserting the processor's INIT# pin.
· By sending the processor an INIT IPI (an IPI with the delivery mode set
to INIT).
Upon receiving an INIT through either of these mechanisms, the processor
responds by beginning the initialization process of the processor core and the
local APIC. The state of the local APIC following an INIT reset is the same as
it is after a power-up or hardware reset, except that the APIC ID and
arbitration ID registers are not affected. This state is also referred to at
the "wait-for-SIPI" state (see also: Section 8.4.2, "MP Initialization
Protocol Requirements and Restrictions").
[...]
Note however that do_cpu_init() does this:
1225 void do_cpu_init(X86CPU *cpu)
1226 {
1227 CPUState *cs = CPU(cpu);
1228 CPUX86State *env = &cpu->env;
1229 CPUX86State *save = g_new(CPUX86State, 1);
1230 int sipi = cs->interrupt_request & CPU_INTERRUPT_SIPI;
1231
1232 *save = *env;
1233
1234 cpu_reset(cs);
1235 cs->interrupt_request = sipi;
1236 memcpy(&env->start_init_save, &save->start_init_save,
1237 offsetof(CPUX86State, end_init_save) -
1238 offsetof(CPUX86State, start_init_save));
1239 g_free(save);
1240
1241 if (kvm_enabled()) {
1242 kvm_arch_do_init_vcpu(cpu);
1243 }
1244 apic_init_reset(cpu->apic_state);
1245 }
The CPU environment, which in this case includes the EFER state, is saved and
restored when calling cpu_reset(). The x86_cpu_reset() function actually does
clear all of the CPU environment, but this function puts it all back.
The result of this is that if the CPU was in long mode and you do an INIT IPI,
the CPU still has the EFER.LMA and EFER.LME bits set, even though it's not
actually running in long mode anymore. It doesn't seem possible for the guest
to get the CPU out of this state, and one nasty side-effect is that trying to
set the CR0 to enable paging never succeeds.
I added the following code at line 1240 above as a workaround:
#ifdef TARGET_X86_64
/*
* The initial state of the CPU is not 64-bit mode. This being
* the case, don't leave the EFER.LME or EFER.LME bits set.
*/
cpu_load_efer(env, 0);
#endif
This seemed to fix the problem I was having, however I'm not certain this is
the correct fix.
As background, I ran across this problem testing VxWorks with QEMU 2.3.0 and
OVMF firmware. The VxWorks BOOTX64.EFI loader is able to load and run 32-bit
VxWorks images on 64-bit hardware by forcing the CPU back to 32-bit mode
before handing control to the OS. However it only does this for the BSP (CPU
0). It turns out that the UEFI firmware puts the AP cores into long mode too.
(This may be new in recent UEFI/OVMF versions, because I'm pretty sure tested
this path before and didn't see a problem.) Everything works ok with
uniprocessor images, but with SMP images, launching the first AP CPU fails due
to the above condition (the CPU starts up, but is unable to enable paging and
dies screaming in short order).
Booting with the 32-bit OVMF build and the VxWorks BOOTIA32.EFI loader works
ok. The same VxWorks loader and kernel code also seems to run ok on real
hardware.
I'm using QEMU 2.3.0 on FreeBSD/amd64 9.2-RELEASE. I'm not using KVM. It looks
like the code is still the same in the git repo. Am I correct that
do_cpu_init() should be clearing the EFER contents?
-Bill
--
=============================================================================
-Bill Paul (510) 749-2329 | Senior Member of Technical Staff,
wp...@windriver.com | Master of Unix-Fu - Wind River Systems
=============================================================================
"I put a dollar in a change machine. Nothing changed." - George Carlin
=============================================================================
