On Tue, Sep 1, 2020 at 1:02 AM Zir Blazer <zir_bla...@hotmail.com> wrote: > > Note that the IOAPIC being used depends on the Chipset. 440FX may use an > optional 82093AA IOAPIC, which is what is in the PDF that you linked earlier > (It used to be a standalone chip and usually only found on Dual Processor > Motherboards. IOAPIC was extremely rare on consumer 440FX Single Processor > Motherboards. QEMU uses it by default because otherwise you wouldn't have > working APIC, thus no SMP/SMT), whereas Q35 includes an IOAPIC integrated in > the Southbridge component. The Q35 Chipset IOAPIC is also more advanced and > provides a total of 24 usable IRQs (The 82093AA has a total of 24, but not > all are usable), check Slide 15-17 here: > https://www.linux-kvm.org/images/0/06/2012-forum-Q35.pdf > So, if you're complaining about the 440FX IOAPIC but didn't tested the Q35 > one already, give it a try.
I think I want the performance of in-kernel IOAPIC emulation. I should go see whether the Q35 IOAPIC emulation has both userspace and kernel space emulation supported or just userspace. Good point though. > > > I agree than it isn't a good idea to arbitrarily increase the amount of IRQs > supported by the current IOAPICs, as you may be breaking something else. I > mean, these IOAPICs are supposed to identify themselves as preexisting chips > with known functions, so they have to mimic the original chips capabilities > so that Software that are intended to interact with them like Drivers behaves > the same in the emulated ones. If you change the emulated IOAPICs > capabilities, they are not a reasonable emulation of the real chips anymore. > Same applies to multiple IOAPICs. Actually, do the 82093AA and Q35 > Southbridge even supported such combination? I know that the APIC > specification allows for multiple IOAPICs, but I doubt these two were > intended for that. Good question, not sure > How would the OS and Chipset Drivers react to such changes? I know that > virtualization has allowed to try physically imposible combinations like > plugging real passthroughed PCIe Devices directly onto an emulated PCI Host > Bridge and other weird combinations and for the most part they worked, so > maybe it is less worse than appears. If you hack (e.g. https://chromium-review.googlesource.com/c/chromiumos/third_party/kernel/+/2250716) the KVM ioapic emulation to support more IRQs, from what I've seen the guest ioapic drivers play along nicely (at least on recent Linux kernel guests). Apparently this is because the guest checks control register content for version, number of pins, few other things rather than hardcoding (or checking ACPI/MADT info). That maybe gives hope for just hacking the in-kernel IOAPIC emulation in kvm. > > > > > > If anything, I'm interesed on whenever you can create from scratch an > emulated function that has no physical Hardware counterpart. It was done some > years ago with the Super Nintendo and the MSU-1 enchantment chip, that was > born in an emulator: > http://nerdlypleasures.blogspot.com/2018/01/sd2snes-and-msu-1.html > I would believe that a custom, virtual platform implementation may be better > in the long run than trying to match the capabilities of existing Hardware > like with the current emulated Chipsets. At the end of the day, emulating > Hardware puts you hard limits of what you should be able to do. I agree > > APIC seems to support some form of detection of one or multiple IOAPICs and > how many IRQs each of them manages via the ACPI MADT Table: > https://wiki.osdev.org/IOAPIC#Detecting_I.2FO_APIC > It may be possible than OS side it uses that to autodetect changes and act > accordingly, so you may throw it something new and get it to work. But I > don't know whenever you can replace with a brand new IOAPIC the one of the > existing Chipsets, or if there are any assumptions by the Firmware/OS/Drivers > side besides the ACPI Table one. > > About old PCI, each PCI Slot has 4 Pins for 4 Interrupt lines (INTA#, INTB#, > INTC#, INTD#). A PCI Device (Or Function?) may only use a single Interrupt, > and a PCI Card with a PCI Device that uses Interrupts always begins using the > INTA# line, the other 3 are used only if there are more Devices that uses > Interrupts. Motherboard side, there are usually 4 lines (LNKA, LNKB, LNKC, > LNKD) that are shared between multiple PCI Slots, usually with a zigzagging > pattern, which coupled with cards always using INTA# first gives the layout > an inherent load balancing. For example, LNKA goes to INTA# in first slot, > then INTD# on second, INTC# on third and INTB# on fourth, and would repeat > INTA# in the fifth slot, LNKB goes to INTB# in first slot, INTA# in second > slot, and so on. > These 4 Motherboard lines end up wired to a PIR (Programmable Interrupt > Router). The PIR is wired to the IOAPIC (Actually, I think that on every > case, the PIR is part of the IOAPIC chip itself), and maps each of the > Motherboard lines to an IOAPIC IRQ. However, a PIR may have more than 4 > inputs, usually 8 (LNKE, LNKF, LNKG, LNKH). You could have an arrangement > where if you have 8 PCI Slots, you could have 4 Motherboard lines (LNKA-D) > take care of the total 16 Interrupt Pins in the first set of 4 slots, then > let the other 4 lines (LNKE-H) take care of all the 16 pins from the other 4 > slots. So, contrary to what AW said, you CAN have more than 4 PCI Interrupts. > > Fore reference, here is a log from a Dell PowerEdge 1655MC that has a massive > 16 Motherboard PCI Interrupt lines (From LNKA to LNKP) using a ServerWorks > ServerSet LE Chipset that has 2 IOAPICs (Not sure if it means that they're > two chips or just logically separated): > http://old-list-archives.xenproject.org/archives/html/xen-users/2006-01/msg00185.html > And here is an ASUS PR-DLS with the ServerWorks Grand Champion LE Chipset > that has a total of 3 IOAPICs and a truckload 32 lines (LNKA to LNK5): > https://sourceforge.net/p/acpi/mailman/message/12179397/ > > > I don't recall anything that said that if you had a wide enough PIR/IOAPIC, > you couldn't get all the 4 Interrupt lines from a single PCI Slot mapped to a > dedicated IRQ each (Heck, you could even eliminate the PIR if going that > way). It isn't done on physical Hardware because the costs would be > ridiculous (Multiply those 4 lines by the amount of Slots, plus two or three > IOAPICs to wire them all, and Motherboard complexity), which is the reason > why PCI has some form of Interrupt load balancing with the zigzagging pattern > of the Interrupt lines between Slots, and the lines are connected to a PIR > instead of the IOAPIC directly, so that you can both balance Interrupt load > and map a Device to a specific IRQ. But since we are talking about > virtualization, I don't see why not go nuts and see if it works. At least on > W2000/XP, it should slighty decrease latency and increase performance if you > don't have to multiplex several Devices onto the same IRQ. Cool thanks for all the info :) > _______________________________________________ > vfio-users mailing list > vfio-users@redhat.com > https://www.redhat.com/mailman/listinfo/vfio-users _______________________________________________ vfio-users mailing list vfio-users@redhat.com https://www.redhat.com/mailman/listinfo/vfio-users
