So I’ve made a little more progress here- Ubuntu 16 - doing the full upgrade with the newest kernel seemed to give me massive performance increase with multiple different configurations. While this has fixed a lot of my issues, it raises even further questions about interrupt management. Switching to the lowlatency kernel also seems to have made a big difference- I believe the linux kernel is more able to keep up with the timer demands of the guest. But at this point I have 0 audio stutter, and latency jitters at +/- 5ms with a frequency of about 20Hz (which is better than I’ve seen on some bare metal streaming hosts!)
The other thing I did was force x2apic mode in windows 10: https://support.microsoft.com/en-us/kb/2303458 So this is where i get a lot of confusion… recent posts are saying to turn of hyper-v APIC because APICv on newer intel is faster (and AFAIK this is done through x2apic emulation). KVM does not seem to enable APICv for me, so I get more stuttering with hyper-v apic off. Now, where the magic really seems to come in…. forcing windows to use X2APIC AND enabling hyper-v gives me the best performance. From what I’ve read in the KVM code, qemu should be giving hints to Windows on what APIC to use. From what i understand, windows should really choose one or the other, but in my case it seems to be using both X2APIC and hyper-v APIC. i think hyper-v APIC is just fast EOI emulation, I can see via kvm_stat that I get paravirtual EOIs with it on, but not with it off. The other magic, is that with X2APIC, I magically don’t have to force MSI interrupts anymore. I get almost identical performance either way (probably some loss for shared edge interrupts with non-msi). So at this point I’m really confused how these different types are being emulated/passed-through. Without x2apic I seem to have no ability to use edge interrupts, but MSI is fine. With x2APIC it all seems… fine, and with really low overhead even though linux tells me I dont have APICv enabled. I posted another thread trying to clear up the difference between the interrupt types: https://www.redhat.com/archives/vfio-users/2016-April/msg00219.html On Thu, Apr 28, 2016 at 10:43 PM Muted Bytes <mutedby...@gmail.com> wrote: > It would be great to get information on all the points your brought up. > Regarding your last few points, specifically > > "Also, how does x2APIC relate to MSI and/or VFIO? I’m overriding my BIOS > opt-out to force x2APIC. Is there any reason to not do this? Does this > interfere with MSI at all? Why would my bios want to disable a much larger > APIC? Can qemu/kvm even utilize this?" > > what I have found so far, a lot of people recently have been suggesting to > remove the hyperv vapic enlightenment so that guests can use APICv hardware > support of more recent Intel CPUs. However, it seems this actually requires > x2apic support in order for guest to actually use apicv. In my case, my > motherboard also requires the optout to be overridden for x2apic to be > enabled. Upon booting with this override option, I have observed a > significant negative performance impact in the host. Browsers lag, and even > ramspeed program shows 15% or worse decrease in ram throughput, among > others. > On Mar 31, 2016 10:57 AM, "Colin Godsey" <crgod...@gmail.com> wrote: > >> I had some questions about IRQ balancing with MSI interrupts from >> passthrough devices using VFIO. >> >> I have a system that is designed to serve as a gaming streaming server >> for 2 windows VMs each using PCIe passthrough. So far the recurring problem >> has been interrupt management. This system ends up being a perfect storm >> (no pun indented) for interrupts, having to manage: high performance timers >> for video and audio, high data transfer from the GPU (more than normal >> because we’re passing 50mbps video stream from the GPU to the system), then >> 50mbps per VM transfer back to the host via virtio networking. I’m giving >> full CPU mapping to both VMs, with staggered affinity (VM0 VCPU0=CPU0, VM1 >> VCPU0=CPU2). It is a non-HT intel SMP CPU. >> >> MSI so far seems to be the key to getting this all to work right. >> Enabling MSI was the first big thing, guests not detecting it led to really >> poor interrupt management, reading more about it explained why. >> >> The low latency kernel seems to be the other part of this- games like to >> run on an accurate clock, and context switching seems minimal when the GPU >> can drive frame composition like normal, and in the end i think it wastes >> less CPU cycles due to accurate media response. Even though this system >> deals with large amounts of throughput, latency and device-responsiveness >> are still mostly preferred. >> >> This leads to the last piece of the puzzle. My mysterious performance >> drop across the board when both systems are under GPU load. Both MSI GPUs >> seem to be piling the majority of their interrupts onto the same CPU. The >> MSI based ethernet adapter seems to fall on another CPU, so that’s great. >> But the GPU seems to just tank overall guest performance and the GPU >> interrupts dont seem to want to budge from CPU1 no matter what. There is >> some distribution of the GPU interrupts across CPUs, but the bulk seems to >> always land on CPU1. >> >> Ultimately what im wondering: >> * How does MSI balance? Via VFIO, are the MSI interrupts >> assigned/mapped/masked on a guest level, host level, or hardware level? >> * Also, how does x2APIC relate to MSI and/or VFIO? I’m overriding my BIOS >> opt-out to force x2APIC. Is there any reason to not do this? Does this >> interfere with MSI at all? Why would my bios want to disable a much larger >> APIC? Can qemu/kvm even utilize this? >> * Could my performance issues just be related to context-switching and >> CPU/cache bounds? Would the low latency kernel (1000Mhz) make that worse? >> If so, is there any way to force MSI affinity? >> >> Thanks tons! >> >> -Colin G >> >> _______________________________________________ >> vfio-users mailing list >> vfio-users@redhat.com >> https://www.redhat.com/mailman/listinfo/vfio-users >> >>
_______________________________________________ vfio-users mailing list vfio-users@redhat.com https://www.redhat.com/mailman/listinfo/vfio-users
