Hi, On 1/9/23 14:24, Eric Auger wrote: > Hi, > > we have a trouble with virtio-iommu and protected assigned devices > downstream to a pcie-to-pci bridge. In that use case we observe the > assigned devices are not put to any group. This is true on both x86 and > aarch64. This use case works with intel-iommu. > > *** Guest PCI topology is: > lspci -tv > -[0000:00]-+-00.0 Intel Corporation 82G33/G31/P35/P31 Express DRAM > Controller > +-01.0 Device 1234:1111 > +-02.0-[01-02]----00.0-[02]----01.0 Broadcom Inc. and > subsidiaries BCM57416 NetXtreme-E Dual-Media 10G RDMA Ethernet Controller > +-02.1-[03]-- > +-02.2-[04]----00.0 Red Hat, Inc. Virtio block device > +-0a.0 Red Hat, Inc. Device 1057 > +-1f.0 Intel Corporation 82801IB (ICH9) LPC Interface Controller > +-1f.2 Intel Corporation 82801IR/IO/IH (ICH9R/DO/DH) 6 port > SATA Controller [AHCI mode] > \-1f.3 Intel Corporation 82801I (ICH9 Family) SMBus Controller > > > All the assigned devices are aliased and they get devfn=0x0. > see qemu pci_device_iommu_address_space in hw/pci.c > > Initially I see the following traces > pci_device_iommu_address_space name=vfio-pci BDF=0x8 bus=0 devfn=0x8 > pci_device_iommu_address_space name=vfio-pci BDF=0x8 bus=0 devfn=0x8 > call iommu_fn with bus=0x55f556dde180 and devfn=0 > virtio_iommu_init_iommu_mr init virtio-iommu-memory-region-0-0 > > Note the bus is 0 at this time and devfn that is used in the > virtio-iommu is 0. So an associated IOMMU MR is created with this bus at > devfn=0 slot. This is before bus actual numbering. > > However later on, I see virtio_iommu_probe() and virtio_iommu_attach() > getting called with ep_id=520 > because in the qemu virtio-iommu device, virtio_iommu_mr(pe_id) fails to > find the iommu_mr and returns -ENOENT > > On guest side I see that > acpi_iommu_configure_id/iommu_probe_device() fails > (__iommu_probe_device) and also __iommu_attach_device would also fail > anyway. > > I guess those get called before actual bus number recomputation? > > on aarch64 I eventually see the "good" MR beeing created, ie. featuring > the right bus number: > qemu-system-aarch64: pci_device_iommu_address_space name=vfio-pci > BDF=0x208 bus=2 devfn=0x8 > qemu-system-aarch64: pci_device_iommu_address_space name=vfio-pci > BDF=0x208 bus=2 devfn=0x8 call iommu_fn with bus=0xaaaaef12c450 and devfn=0 > > But this does not happen on x86. > > Jean, do you have any idea about how to fix that? Do you think we have a > trouble in the acpi/viot setup or virtio-iommu probe sequence. It looks > like virtio probe and attach commands are called too early, before the > bus is actually correctly numbered.
So after further investigations looks this is not a problem of bus number, which is good at the time of the virtio cmd calls but rather a problem related to the devfn (0 was used when creating the IOMMU MR) whereas the virtio-iommu cmds looks for the non aliased devfn. With that fixed, the probe and attach at least succeeds. The device still does not work for me but I will continue my investigations and send a tentative fix. Thanks Eric > > Thanks > > Eric > > > > > > >
