Hi Shameer, On 11/20/24 15:16, Shameerali Kolothum Thodi wrote: > >> -----Original Message----- >> From: Eric Auger <eric.au...@redhat.com> >> Sent: Monday, November 18, 2024 6:10 PM >> To: Shameerali Kolothum Thodi >> <shameerali.kolothum.th...@huawei.com>; qemu-...@nongnu.org; >> qemu-devel@nongnu.org >> Cc: peter.mayd...@linaro.org; j...@nvidia.com; nicol...@nvidia.com; >> ddut...@redhat.com; Linuxarm <linux...@huawei.com>; Wangzhou (B) >> <wangzh...@hisilicon.com>; jiangkunkun <jiangkun...@huawei.com>; >> Jonathan Cameron <jonathan.came...@huawei.com>; >> zhangfei....@linaro.org >> Subject: Re: [RFC PATCH 4/5] hw/arm/virt-acpi-build: Build IORT with >> multiple SMMU nodes > [...] > >>> I think the above won't affect the basic case where I have only one >>> pcie-pxb/SMMUv3. But even in that case hot add seems not working. >>> >>> I tried hacking the min/max ranges as suspected by Nicolin. But still not >> enough to >>> get it working. Do you have any hint on why the hot add(described >> below) is not >>> working? >> Hum thought the duplicate idmap could be the cause. Otherwise I have no >> clue. I would advice to fix it first. > I think I have an idea why the hot add was not working. > > When we have the PCIe topology as something like below, > > -device pxb-pcie,id=pcie.1,bus_nr=8,bus=pcie.0 \ > -device pcie-root-port,id=pcie.port1,bus=pcie.1,chassis=1 \ > -device pcie-root-port,id=pcie.port2,bus=pcie.1,chassis=2 \ > -device arm-smmuv3-nested,id=smmuv1,pci-bus=pcie.1 \ > ... > > The current IORT generation includes the pcie-root-port dev ids also > in the SMMUv3 node idmaps. > > Hence, when Guest kernel loads, pcieport is also behind the SMMUv3. > > [ 1.466670] pcieport 0000:64:00.0: Adding to iommu group 1 > ... > [ 1.448205] pcieport 0000:64:01.0: Adding to iommu group 2
But it should be the same without multi-instantiation, no? I would have expected this as normal. Has you tested hot-plug without the series laterly? Do you have the same pb? Thanks Eric > > > So when we do a hot add, > device_add vfio-pci,host=0000:75:00.1,bus=pcie.port1,iommufd=iommufd0 > > The Qemu hotplug event handler tries to inject an IRQ to the Guest pcieport > by retrieving the MSI address it is configured with. > > hotplug_event_notify() > msix_prepare_message(): [address: 0xfffff040] > msix_notify() > > The ITS address retrieved here is actually the SMMUv3 translated iova addr, > not the Guest PA. So Guest never sees/receives the interrupt. > > I did hack the IORT code to exclude the pcie-root-port dev ids from the SMMUv3 > node idmaps and the hot add seems to work fine. > > Looks like we need to find all the pcie-root-port dev ids associated with a > SMMUv3/pxb-pcie and exclude them from SMMUv3 node idmaps to get > the hot add working. > > I am not sure though this will create any other issues in IOMMU isolation > criteria > (ACS etc,), especially if we want to access the device in Guest user space( I > hope > not). > > Thanks, > Shameer > > > >
