On 13/03/2025 4:07 pm, Jan Beulich wrote: > On 13.03.2025 17:02, Andrew Cooper wrote: >> On 13/03/2025 3:47 pm, Roger Pau Monné wrote: >>> On Thu, Mar 13, 2025 at 11:30:28AM -0400, Jason Andryuk wrote: >>>> On 2025-02-27 05:23, Roger Pau Monné wrote: >>>>> On Wed, Feb 26, 2025 at 04:11:25PM -0500, Jason Andryuk wrote: >>>>>> The ath11k device supports and tries to enable 32 MSIs. Linux in PVH >>>>>> dom0 and HVM domU fails enabling 32 and falls back to just 1, so that is >>>>>> all that has been tested. >>>>> DYK why it fails to enable 32? >>>> In Linux msi_capability_init() >>>> >>>> /* Reject multi-MSI early on irq domain enabled architectures */ >>>> if (nvec > 1 && !pci_msi_domain_supports(dev, >>>> MSI_FLAG_MULTI_PCI_MSI, ALLOW_LEGACY)) >>>> return 1; >>>> >>>> MSI_FLAG_MULTI_PCI_MSI is only set for AMD and Intel interrupt remapping, >>>> and Xen PVH and HVM don't have either of those. They are using "VECTOR", >>>> so >>>> this check fails. >>> Oh, interesting. So classic PV MSI domain supports >>> MSI_FLAG_MULTI_PCI_MSI, even when no IOMMU is exposed there either. >>> >>> Thanks, so it's nothing specific to Xen, just how Linux works. >> This is something which TGLX and I have discussed in the past. It is a >> mistake for any x86 system to do MSI multi-message without an IOMMU. > Well, with PVH there always will be an IOMMU, just that Linux can't see > it. Even with PV it should be the hypervisor to determine whether multi- > message MSI is possible. Hence how the classic (non-pvops) kernel had > worked in this regard.
Xen should hide (and instruct Qemu to hide) multi-message on non-IOMMU hardware. The result of "supporting" them on non-IOMMU hardware is worse than making the driver run in single MSI mode. While in theory Xen could expose "I've got an IOMMU so you can do multi message" to guests, this isn't trivial for the guest to cope with. Even if the guest knows multi-message is safe, it still cant express this to Xen via a multi-message shaped interface. With Teddy's PV-IOMMU, this problem ought to go away because now the guest can see the IOMMU. ~Andrew
