> -----Original Message-----
> From: Eric Auger <eric.au...@redhat.com>
> Sent: Monday, November 18, 2024 6:10 PM
> To: Shameerali Kolothum Thodi
> <shameerali.kolothum.th...@huawei.com>; qemu-...@nongnu.org;
> qemu-devel@nongnu.org
> Cc: peter.mayd...@linaro.org; j...@nvidia.com; nicol...@nvidia.com;
> ddut...@redhat.com; Linuxarm <linux...@huawei.com>; Wangzhou (B)
> <wangzh...@hisilicon.com>; jiangkunkun <jiangkun...@huawei.com>;
> Jonathan Cameron <jonathan.came...@huawei.com>;
> zhangfei....@linaro.org
> Subject: Re: [RFC PATCH 4/5] hw/arm/virt-acpi-build: Build IORT with
> multiple SMMU nodes
[...]
> > I think the above won't affect the basic case where I have only one
> > pcie-pxb/SMMUv3. But even in that case hot add seems not working.
> >
> > I tried hacking the min/max ranges as suspected by Nicolin. But still not
> enough to
> > get it working. Do you have any hint on why the hot add(described
> below) is not
> > working?
> Hum thought the duplicate idmap could be the cause. Otherwise I have no
> clue. I would advice to fix it first.
I think I have an idea why the hot add was not working.
When we have the PCIe topology as something like below,
-device pxb-pcie,id=pcie.1,bus_nr=8,bus=pcie.0 \
-device pcie-root-port,id=pcie.port1,bus=pcie.1,chassis=1 \
-device pcie-root-port,id=pcie.port2,bus=pcie.1,chassis=2 \
-device arm-smmuv3-nested,id=smmuv1,pci-bus=pcie.1 \
...
The current IORT generation includes the pcie-root-port dev ids also
in the SMMUv3 node idmaps.
Hence, when Guest kernel loads, pcieport is also behind the SMMUv3.
[ 1.466670] pcieport 0000:64:00.0: Adding to iommu group 1
...
[ 1.448205] pcieport 0000:64:01.0: Adding to iommu group 2
So when we do a hot add,
device_add vfio-pci,host=0000:75:00.1,bus=pcie.port1,iommufd=iommufd0
The Qemu hotplug event handler tries to inject an IRQ to the Guest pcieport
by retrieving the MSI address it is configured with.
hotplug_event_notify()
msix_prepare_message(): [address: 0xfffff040]
msix_notify()
The ITS address retrieved here is actually the SMMUv3 translated iova addr,
not the Guest PA. So Guest never sees/receives the interrupt.
I did hack the IORT code to exclude the pcie-root-port dev ids from the SMMUv3
node idmaps and the hot add seems to work fine.
Looks like we need to find all the pcie-root-port dev ids associated with a
SMMUv3/pxb-pcie and exclude them from SMMUv3 node idmaps to get
the hot add working.
I am not sure though this will create any other issues in IOMMU isolation
criteria
(ACS etc,), especially if we want to access the device in Guest user space( I
hope
not).
Thanks,
Shameer
