On 05/10/18 11:48, Wei Liu wrote:
> Got this one-off crash while booting staging (d36b770458) on a skylake
> server. After rebooting it went away.
>
> (XEN) Assertion '(sp == 0) || (peoi[sp-1].vector < vector)' failed at
> irq.c:1173
> (XEN) ----[ Xen-4.12-unstable x86_64 debug=y Tainted: C ]----
>
> (XEN) CPU: 5
>
> (XEN) RIP: e008:[<ffff82d080286921>] do_IRQ+0x496/0x680
>
> (XEN) RFLAGS: 0000000000010046 CONTEXT: hypervisor
>
> (XEN) rax: ffff83085df7a4c0 rbx: ffff83085df81e00 rcx: 0000000000000001
>
> (XEN) rdx: 0000000000000021 rsi: 0000000000000021 rdi: 0000000000000001
>
> (XEN) rbp: ffff83085df77d98 rsp: ffff83085df77d38 r8: 0000000000000021
>
> (XEN) r9: 0000000000000000 r10: 0000000000000000 r11: 0000000000000000
>
> (XEN) r12: ffff8308740e4f10 r13: 0000000000000021 r14: ffff83085df81e00
>
> (XEN) r15: 000000000000001e cr0: 0000000080050033 cr4: 00000000003526e0
>
> (XEN) cr3: 000000085da9e000 cr2: 00007fc5b6a8cfe8
>
> (XEN) fsb: 0000000000000000 gsb: 0000000000000000 gss: 0000000000000000
>
> (XEN) ds: 0000 es: 0000 fs: 0000 gs: 0000 ss: 0000 cs: e008
>
> (XEN) Xen code around <ffff82d080286921> (do_IRQ+0x496/0x680):
>
> (XEN) be 00 00 00 7e 93 0f 0b <0f> 0b 0f 0b 0f 0b b8 00 00 00 00 eb 4e 83 bb
> 1c
> (XEN) Xen stack trace from rsp=ffff83085df77d38:
>
> (XEN) ffff82d000000000 ffff83085df81e24 0000000000000000 0000001e8037a835
>
> (XEN) ffff82d08037a841 ffff82d08037a835 ffff82d08037a841 0000000000000000
>
> (XEN) 0000000000000000 0000000000000000 ffff83085df77fff 0000000000000000
>
> (XEN) 00007cf7a2088237 ffff82d08037a8aa 0000000380f0b241 0000000000000008
>
> (XEN) ffff83085df79448 ffff83085df79390 ffff83085df77ec0 0000000380f62e26
>
> (XEN) 00000003810cc680 ffff8307de5670a8 00000000001f644f 0000000000000809
>
> (XEN) ffff83085df7a02c 0000000000000000 ffff83085df77fff 00000000000051f3
>
> (XEN) ffff83085df793c0 0000002100000000 ffff82d0802e1684 000000000000e008
>
> (XEN) 0000000000000202 ffff83085df77e50 0000000000000000 ffff82d08059bc80
>
> (XEN) 00000020ffffffff ffff83085df77fff ffff82d0805a3c80 ffff83085df77eb0
>
> (XEN) 0000000000000000 0000000000000000 0000033b00000212 ffff82d08059bf00
>
> (XEN) 0000000000000005 ffff82d08059bf00 0000000000000005 0000000000000005
>
> (XEN) ffff83085df39000 ffff83085df77ef0 ffff82d0802770b8 ffff830864159000
>
> (XEN) ffff8300791fd000 ffff8300791fb000 ffff830864159000 ffff83085df77db8
>
> (XEN) 0000000000000000 0000000000000000 ffff88017dbe3d00 ffff88017dbe3d00
>
> (XEN) 0000000000000002 0000000000000002 0000000000000000 0000000000000000
>
> (XEN) 0000000148106000 000000005236fe09 ffffffff816fe980 ffff880182a9a7c0
>
> (XEN) ffffffff82049af8 ffff880182a9a7c0 0000000000000082 0000beef0000beef
>
> (XEN) ffffffff816fec52 000000bf0000beef 0000000000000246 ffffc900006dbe98
>
> (XEN) 000000000000beef 000000000000beef 000000000000beef 000000000000beef
>
> (XEN) Xen call trace:
>
> (XEN) [<ffff82d080286921>] do_IRQ+0x496/0x680
>
> (XEN) [<ffff82d08037a8aa>] common_interrupt+0x10a/0x120
>
> (XEN) [<ffff82d0802e1684>] mwait-idle.c#mwait_idle+0x296/0x372
>
> (XEN) [<ffff82d0802770b8>] domain.c#idle_loop+0xb3/0xb5
>
> (XEN)
>
> (XEN)
>
> (XEN) ****************************************
>
> (XEN) Panic on CPU 5:
>
> (XEN) Assertion '(sp == 0) || (peoi[sp-1].vector < vector)' failed at
> irq.c:1173
> (XEN) ****************************************
>
> (XEN)
>
> (XEN) Manual reset required ('noreboot' specified)
>
> Let me know what else is needed.
We've seen this reported sporadically in the past, and never with enough
information to investigate. I had one reliable repro of the issue in
the past, which disappeared with a microcode update.
It is always out of mwait, and Xen's logic for which interrupts are
pending now disagrees with hardware. This means we've seen an interrupt
at a lower priority that one we believe to be pending, which is (to a
first approximation), a violation of LAPIC priority logic.
Perhaps I should insert a tonne of debugging in place of this assertion,
with the hope that the next time we randomly encounter it, we've got
some better idea of what is going on.
~Andrew
_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xenproject.org
https://lists.xenproject.org/mailman/listinfo/xen-devel
