On 14/04/2022 22:31, Cédric Le Goater wrote:
Hello Alexey, Thanks for taking over. On 4/13/22 06:56, Alexey Kardashevskiy wrote:On 3/17/22 06:16, Cédric Le Goater wrote:Timothy, On 3/16/22 17:29, Cédric Le Goater wrote:Hello,I've been struggling for some time with what is looking like a potential bug in QEMU/KVM on the POWER9 platform. It appears that in XIVE mode, when the in-kernel IRQ chip is enabled, an external device that rapidly asserts IRQs via the legacy INTx level mechanism will only receive one interrupt in the KVM guest.Indeed. I could reproduce with a pass-through PCI adapter using 'pci=nomsi'. The virtio devices operate correctly but the network adapter only receives one event (*): $ cat /proc/interruptsCPU0 CPU1 CPU2 CPU3 CPU4 CPU5 CPU6 CPU7 16: 2198 1378 1519 1216 0 0 0 0 XIVE-IPI 0 Edge IPI-0 17: 0 0 0 0 2003 1936 1335 1507 XIVE-IPI 1 Edge IPI-1 18: 0 6401 0 0 0 0 0 0 XIVE-IRQ 4609 Level virtio3, virtio0, virtio2 19: 0 0 0 0 0 204 0 0 XIVE-IRQ 4610 Level virtio1 20: 0 0 0 0 0 0 0 0 XIVE-IRQ 4608 Level xhci-hcd:usb1 21: 0 1 0 0 0 0 0 0 XIVE-IRQ 4612 Level eth1 (*) 23: 0 0 0 0 0 0 0 0 XIVE-IRQ 4096 Edge RAS_EPOW 24: 0 0 0 0 0 0 0 0 XIVE-IRQ 4592 Edge hvc_console 26: 0 0 0 0 0 0 0 0 XIVE-IRQ 4097 Edge RAS_HOTPLUGChanging any one of those items appears to avoid the glitch, e.g. XICSXICS is very different from XIVE. The driver implements the previous interrupt controller architecture (P5-P8) and the hypervisor mediates the delivery to the guest. With XIVE, vCPUs are directly signaled by the IC. When under KVM, we use different KVM devices for each mode : * KVM XIVE is a XICS-on-XIVE implementation (P9/P10 hosts) for guests not using the XIVE native interface. RHEL7 for instance. * KVM XIVE native is a XIVE implementation (P9/P10 hosts) for guests using the XIVE native interface. Linux > 4.14. * KVM XICS is for P8 hosts (no XIVE HW) VFIO adds some complexity with the source events. I think the problem comes from the assertion state. I will talk about it later.mode with the in-kernel IRQ chip works (all interrupts are passed through),All interrupts are passed through using XIVE also. Run 'info pic' in the monitor. On the host, check the IRQ mapping in the debugfs file : /sys/kernel/debug/powerpc/kvm-xive-*and XIVE mode with the in-kernel IRQ chip disabled also works.In that case, no KVM device backs the QEMU device and all state is in one place.We are also not seeing any problems in XIVE mode with the in-kernel chip from MSI/MSI-X devices.Yes. pass-through devices are expected to operate correctly :)The device in question is a real time card that needs to raise an interrupt every 1ms. It works perfectly on the host, but fails in the guest -- with the in-kernel IRQ chip and XIVE enabled, it receives exactly one interrupt, at which point the host continues to see INTx+ but the guest sees INTX-, and the IRQ handler in the guest kernel is never reentered.ok. Same symptom as the scenario above.We have also seen some very rare glitches where, over a long period of time, we can enter a similar deadlock in XICS mode.with the in-kernel XICS IRQ chip ?Disabling the in-kernel IRQ chip in XIVE mode will also lead to the lockup with this device, since the userspace IRQ emulation cannot keep up with the rapid interrupt firing (measurements show around 100ms required for processing each interrupt in the user mode).MSI emulation in QEMU is slower indeed (35%). LSI is very slow because it is handled as a special case in the device/driver. To maintain the assertion state, all LSI handling is done with a special HCALL : H_INT_ESB which is implemented in QEMU. This generates a lot of back and forth in the KVM stack.My understanding is the resample mechanism does some clever tricks with level IRQs, but that QEMU needs to check if the IRQ is still asserted by the device on guest EOI.Yes. the problem is in that area.Since a failure here would explain these symptoms I'm wondering if there is a bug in either QEMU or KVM for POWER / pSeries (SPAPr) where the IRQ is not resampled and therefore not re-fired in the guest?KVM I would say. The assertion state is maintained in KVM for the KVM XICS-on-XIVE implementation and in QEMU for the KVM XIVE native device. These are good candidates. I will take a look.All works fine with KVM_CAP_IRQFD_RESAMPLE=false in QEMU. Can you pleasetry this workaround for now ? I could reach 934 Mbits/sec on the passthrudevice. I clearly overlooked that part and it has been 3 years.Disabling KVM_CAP_IRQFD_RESAMPLE on XIVE-capable machines seems to be the right fix actually.XIVE == baremetal/vm POWER9 and newer.XICS == baremetal/vm POWER8 and older, or VMs on any POWER (backward compat.).yes. You can force XICS on POWER9 using 'max-cpu-compat' or 'ic-mode'.Tested on POWER9 with a passed through XHCI host and "-append pci=nomsi" and "-machine pseries,ic-mode=xics,kernel_irqchip=on" (and s/xics/xive/).ok. This is deactivating the default XIVE (P9+) mode at the platform level,hence forcing the XICS (P8) mode in a POWER9 guest running on a POWER9 host.It is also deactivating MSI, forcing INTx usage in the kernel and forcingthe use of the KVM irqchip device to make sure we are not emulating in QEMU.We are far from the default scenario but this is it !
well, "-machine pseries,ic-mode=xive,kernel_irqchip=on" is the default. "pci=nomsi" is not but since that actual device is only capable on INTx, the default settings expose the problem.
When it is XIVE-on-XIVE (host and guest are XIVE),We call this mode : XIVE native, or exploitation, mode. Anyhow, it is alwaysXIVE under the hood on a POWER9/POWER10 box.INTx is emulated in the QEMU's H_INT_ESB handlerLSI are indeed all handled at the QEMU level with the H_INT_ESB hcall. If I remember well, this is because we wanted a simple way to synthesize the interrupt trigger upon EOI when the level is still asserted. Doing this way is compatible for both kernel_irqchip=off/on modes because the level is maintained in QEMU. This is different for the other two XICS KVM devices which maintain the assertion level in KVM.and IRQFD_RESAMPLE is just useless in such case (as it is designed to eliminate going to the userspace for the EOI->INTx unmasking) and there is no pathway to call the eventfd's irqfd_resampler_ack() from QEMU. So the VM's XHCI device receives exactly 1 interrupt and that is it. "kernel_irqchip=off" fixes it (obviously).yes.When it is XICS-on-XIVE (host is XIVE and guest is XICS),yes (FYI, we have similar glue in skiboot ...)then the VM receives 100000 interrupts and then it gets frozen (__report_bad_irq() is called). Which happens because (unlike XICS-on-XICS), the host XIVE's xive_(rm|vm)_h_eoi() does not call irqfd_resampler_ack(). This fixes it:=============diff --git a/arch/powerpc/kvm/book3s_xive_template.c b/arch/powerpc/kvm/book3s_xive_template.cindex b0015e05d99a..9f0d8e5c7f4b 100644 --- a/arch/powerpc/kvm/book3s_xive_template.c +++ b/arch/powerpc/kvm/book3s_xive_template.c@@ -595,6 +595,8 @@ X_STATIC int GLUE(X_PFX,h_eoi)(struct kvm_vcpu *vcpu, unsigned long xirr)xc->hw_cppr = xc->cppr; __x_writeb(xc->cppr, __x_tima + TM_QW1_OS + TM_CPPR); + kvm_notify_acked_irq(vcpu->kvm, 0, irq); + return rc; } =============OK. XICS-on-XIVE is also broken then :/ what about XIVE-on-XIVE ?
Not sure I am following (or you are) :) INTx is broken on P9 in either mode. MSI works in both.
The host's XICS does call kvm_notify_acked_irq() (I did not test that but the code seems to be doing so).After re-reading what I just wrote, I am leaning towards disabling use of KVM_CAP_IRQFD_RESAMPLE as it seems last worked on POWER8 and never since :)and it would fix XIVE-on-XIVE. Are you saying that passthru on POWER8 is broken ? fully or only INTx ?
No, the opposite - P8 works fine, kvm_notify_acked_irq() is there.
Did I miss something in the picture (hey Cedric)?You seem to have all combination in mind: host OS, KVM, QEMU, guest OS For the record, here is a documentation we did: https://qemu.readthedocs.io/en/latest/specs/ppc-spapr-xive.html It might need some updates.
When I read this, a quite from the Simpsons pops up in my mind: “Dear Mr. President there are too many states nowadays. Please eliminate three. I am NOT a crackpot.” :)
Thanks, C.
