Hi Peter, On 2/28/20 5:14 PM, Peter Xu wrote: > v2: > - pick tags > - don't register resamplefd with KVM kernel when the userspace > resamplefd path is enabled (should enable fast path on new kernels) > - fix resamplefd mem leak > - fix commit message of patch 4 [Eric] > - let kvm_resample_fd_notify() return a boolean, skip ioapic check if > returned true > - more comments here and there in the code to state the fact that > userspace ioapic irr & remote-irr are bypassed [Paolo] > > VFIO INTx is not working with split irqchip. On new kernels KVM_IRQFD > will directly fail with resamplefd attached so QEMU will automatically > fallback to the INTx slow path. However on old kernels it's still > broken. > > Only until recently I noticed that this could also break PXE boot for > assigned NICs [1]. My wild guess is that the PXE ROM will be mostly > using INTx as well, which means we can't bypass that even if we > enables MSI for the guest kernel. > > This series tries to first fix this issue function-wise, then speed up > for the INTx again with resamplefd (mostly following the ideas > proposed by Paolo one year ago [2]). My TCP_RR test shows that: > > - Before this series: this is broken, no number to show > > - After patch 1 (enable slow path): get 63% perf comparing to full > kernel irqchip > > - After whole series (enable fast path partly, irq injection will be > the same as fast path, however userspace needs to intercept for > EOI broadcast to resamplefd, though should still be faster than > the MMIO trick for intx eoi): get 93% perf comparing to full > kernel irqchip, which is a 46% performance boost > > I think we can consider to apply patch 1 even sooner than the rest of > the series to unbreak intx+split first. > > The whole test matrix for reference: > > > |----------+---------+-------------------+--------------+--------------------| > | IRQ type | irqchip | TCP_STREAM (Gbps) | TCP_RR (pps) | note > | > > |----------+---------+-------------------+--------------+--------------------| > | msi | on | 9.39 | 17567 | > | > | nomsi | on | 9.29 | 14056 | > | > | msi | split | 9.36 | 17330 | > | > | nomsi | split | / | / | currently broken > | > | nomsi | split | 8.98 | 8977 | after patch 1 > | > | nomsi | split | 9.21 | 13142 | after whole > series | > > |----------+---------+-------------------+--------------+--------------------| > > Any review comment is welcomed. Thanks, > > [1] https://bugzilla.redhat.com/show_bug.cgi?id=1786404 > [2] https://patchwork.kernel.org/patch/10738541/#22609933
For the whole series: Tested-by: Eric Auger <eric.au...@redhat.com> using a 5.2-rc1 kernel with reverted "654f1f13ea56 kvm: Check irqchip mode before assign irqfd" and guest booting with nomsi. Thanks Eric > > Peter Xu (5): > vfio/pci: Disable INTx fast path if using split irqchip > vfio/pci: Use kvm_irqchip_add_irqfd_notifier_gsi() for irqfds > KVM: Pass EventNotifier into kvm_irqchip_assign_irqfd > KVM: Kick resamplefd for split kernel irqchip > Revert "vfio/pci: Disable INTx fast path if using split irqchip" > > accel/kvm/kvm-all.c | 101 +++++++++++++++++++++++++++++++++++++---- > accel/kvm/trace-events | 1 + > hw/intc/ioapic.c | 23 +++++++++- > hw/vfio/pci.c | 37 ++++++--------- > include/sysemu/kvm.h | 7 +++ > 5 files changed, 137 insertions(+), 32 deletions(-) >
