> -----Original Message----- > From: Daniel P. BerrangĂ© <berra...@redhat.com> > Sent: Thursday, February 6, 2025 2:47 PM > To: Shameerali Kolothum Thodi <shameerali.kolothum.th...@huawei.com> > Cc: qemu-...@nongnu.org; qemu-devel@nongnu.org; > eric.au...@redhat.com; peter.mayd...@linaro.org; j...@nvidia.com; > nicol...@nvidia.com; ddut...@redhat.com; Linuxarm > <linux...@huawei.com>; Wangzhou (B) <wangzh...@hisilicon.com>; > jiangkunkun <jiangkun...@huawei.com>; Jonathan Cameron > <jonathan.came...@huawei.com>; zhangfei....@linaro.org; > nath...@nvidia.com > Subject: Re: [RFC PATCH 0/5] hw/arm/virt: Add support for user-creatable > nested SMMUv3 > > On Thu, Feb 06, 2025 at 01:51:15PM +0000, Shameerali Kolothum Thodi > wrote: > > Hmm..I donât think just swapping the order will change the association > with > > Guest SMMU here. Because, we have, > > > > > -device arm-smmuv3-accel,id=smmuv2,bus=pcie.2 > > > > During smmuv3-accel realize time, this will result in, > > pci_setup_iommu(primary_bus, ops, smmu_state); > > > > And when the vfio dev realization happens, > > set_iommu_device() > > smmu_dev_set_iommu_device(bus, smmu_state, ,) > > --> this is where the guest smmuv3-->host smmuv3 association is first > > established. And any further vfio dev to this Guest SMMU will > > only succeeds if it belongs to the same phys SMMU. > > > > ie, the Guest SMMU to pci bus association, actually make sure you have > the > > same Guest SMMU for the device. > > Ok, so at time of VFIO device realize, QEMU is telling the kernel > to associate a physical SMMU, and its doing this with the virtual > SMMU attached to PXB parenting the VFIO device. > > > smmuv2 --> pcie.2 --> (pxb-pcie, numa_id = 1) > > 0000:dev2 --> pcie.port2 --> pcie.2 --> smmuv2 (pxb-pcie, numa_id = 1) > > > > Hence the association of 0000:dev2 to Guest SMMUv2 remain same. > > Yes, I concur the SMMU physical <-> virtual association should > be fixed, as long as the same VFIO device is always added to > the same virtual SMMU. > > > I hope this is clear. And I am not sure the association will be broken in > any > > other way unless Qemu CLI specify the dev to a different PXB. > > Although the ordering is at least predictable, I remain uncomfortable > about the idea of the virtual SMMU association with the physical SMMU > being a side effect of the VFIO device placement. > > There is still the open door for admin mis-configuration that will not > be diagnosed. eg consider we attached VFIO device 1 from the host NUMA > node 1 to a PXB associated with host NUMA node 0. As long as that's > the first VFIO device, the kernel will happily associate the physical > and guest SMMUs.
Yes. A mis-configuration can place it on a wrong one. > If we set the physical/guest SMMU relationship directly, then at the > time the VFIO device is plugged, we can diagnose the incorrectly > placed VFIO device, and better reason about behaviour. Agree. > I've another question about unplug behaviour.. > > 1. Plug a VFIO device for host SMMU 1 into a PXB with guest SMMU 1. > => Kernel associates host SMMU 1 and guest SMMU 1 together > 2. Unplug this VFIO device > 3. Plug a VFIO device for host SMMU 2 into a PXB with guest SMMU 1. > > Does the host/guest SMMU 1<-> 1 association remain set after step 2, > implying step 3 will fail ? Or does it get unset, allowing step 3 > to succeed, and establish a new mapping host SMMU 2 to guest SMMU 1. At the moment the first association is not persistent. So a new mapping is possible. > If step 2 does NOT break the association, do we preserve that > across a savevm+loadvm sequence of QEMU. If we don't, then step > 3 would fail before the savevm, but succeed after the loadvm. Right. I haven't attempted migration tests yet. But agree that an explicit association is better to make migration compatible. Also I am not sure if the target has a different phys SMMUV3<--> dev mapping how we handle that. > Explicitly representing the host SMMU association on the guest SMMU > config makes this behaviour unambiguous. The host / guest SMMU > relationship is fixed for the lifetime of the VM and invariant of > whatever VFIO device is (or was previously) plugged. > > So I still go back to my general principle that automatic side effects > are an undesirable idea in QEMU configuration. We have a long tradition > of making everything entirely explicit to produce easily predictable > behaviour. Ok. Convinced đ. Thanks for explaining. Shameer
