Hi Shameer,
On 1/31/25 10:33 AM, Shameerali Kolothum Thodi wrote: > >> -----Original Message----- >> From: Shameerali Kolothum Thodi >> Sent: Thursday, January 30, 2025 6:09 PM >> To: 'Daniel P. Berrangé' <berra...@redhat.com> >> Cc: qemu-...@nongnu.org; qemu-devel@nongnu.org; >> eric.au...@redhat.com; peter.mayd...@linaro.org; j...@nvidia.com; >> nicol...@nvidia.com; ddut...@redhat.com; Linuxarm >> <linux...@huawei.com>; Wangzhou (B) <wangzh...@hisilicon.com>; >> jiangkunkun <jiangkun...@huawei.com>; Jonathan Cameron >> <jonathan.came...@huawei.com>; zhangfei....@linaro.org >> Subject: RE: [RFC PATCH 0/5] hw/arm/virt: Add support for user-creatable >> nested SMMUv3 >> >> Hi Daniel, >> >>> -----Original Message----- >>> From: Daniel P. Berrangé <berra...@redhat.com> >>> Sent: Thursday, January 30, 2025 4:00 PM >>> To: Shameerali Kolothum Thodi >> <shameerali.kolothum.th...@huawei.com> >>> Cc: qemu-...@nongnu.org; qemu-devel@nongnu.org; >>> eric.au...@redhat.com; peter.mayd...@linaro.org; j...@nvidia.com; >>> nicol...@nvidia.com; ddut...@redhat.com; Linuxarm >>> <linux...@huawei.com>; Wangzhou (B) <wangzh...@hisilicon.com>; >>> jiangkunkun <jiangkun...@huawei.com>; Jonathan Cameron >>> <jonathan.came...@huawei.com>; zhangfei....@linaro.org >>> Subject: Re: [RFC PATCH 0/5] hw/arm/virt: Add support for user-creatable >>> nested SMMUv3 >>> >>> On Fri, Nov 08, 2024 at 12:52:37PM +0000, Shameer Kolothum via wrote: >>>> How to use it(Eg:): >>>> >>>> On a HiSilicon platform that has multiple physical SMMUv3s, the ACC >> ZIP >>> VF >>>> devices and HNS VF devices are behind different SMMUv3s. So for a >>> Guest, >>>> specify two smmuv3-nested devices each behind a pxb-pcie as below, >>>> >>>> ./qemu-system-aarch64 -machine virt,gic-version=3,default-bus-bypass- >>> iommu=on \ >>>> -enable-kvm -cpu host -m 4G -smp cpus=8,maxcpus=8 \ >>>> -object iommufd,id=iommufd0 \ >>>> -bios QEMU_EFI.fd \ >>>> -kernel Image \ >>>> -device virtio-blk-device,drive=fs \ >>>> -drive if=none,file=rootfs.qcow2,id=fs \ >>>> -device pxb-pcie,id=pcie.1,bus_nr=8,bus=pcie.0 \ >>>> -device pcie-root-port,id=pcie.port1,bus=pcie.1,chassis=1 \ >>>> -device arm-smmuv3-nested,id=smmuv1,pci-bus=pcie.1 \ >>>> -device vfio-pci,host=0000:7d:02.1,bus=pcie.port1,iommufd=iommufd0 \ >>>> -device pxb-pcie,id=pcie.2,bus_nr=16,bus=pcie.0 \ >>>> -device pcie-root-port,id=pcie.port2,bus=pcie.2,chassis=2 \ >>>> -device arm-smmuv3-nested,id=smmuv2,pci-bus=pcie.2 \ >>>> -device vfio-pci,host=0000:75:00.1,bus=pcie.port2,iommufd=iommufd0 \ >>>> -append "rdinit=init console=ttyAMA0 root=/dev/vda2 rw >>> earlycon=pl011,0x9000000" \ >>>> -device virtio-9p-pci,fsdev=p9fs2,mount_tag=p9,bus=pcie.0 \ >>>> -fsdev local,id=p9fs2,path=p9root,security_model=mapped \ >>>> -net none \ >>>> -nographic >>> Above you say the host has 2 SMMUv3 devices, and you've created 2 >>> SMMUv3 >>> guest devices to match. >>> >>> The various emails in this thread & libvirt thread, indicate that each >>> guest SMMUv3 is associated with a host SMMUv3, but I don't see any >>> property on the command line for 'arm-ssmv3-nested' that tells it which >>> host eSMMUv3 it is to be associated with. >>> >>> How does this association work ? >> You are right. The association is not very obvious in Qemu. The association >> and checking is done implicitly by kernel at the moment. I will try to >> explain >> it here. >> >> Each "arm-smmuv3-nested" instance, when the first device gets attached >> to it, will create a S2 HWPT and a corresponding SMMUv3 domain in kernel >> SMMUv3 driver. This domain will have a pointer representing the physical >> SMMUv3 that the device belongs. And any other device which belongs to >> the same physical SMMUv3 can share this S2 domain. >> >> If a device that belongs to a different physical SMMUv3 gets attached to >> the above domain, the HWPT attach will eventually fail as the physical >> smmuv3 in the domains will have a mismatch, >> https://elixir.bootlin.com/linux/v6.13/source/drivers/iommu/arm/arm- >> smmu-v3/arm-smmu-v3.c#L2860 >> >> And as I mentioned in cover letter, Qemu will report, >> >> " >> Attempt to add the HNS VF to a different SMMUv3 will result in, >> >> -device vfio-pci,host=0000:7d:02.2,bus=pcie.port3,iommufd=iommufd0: >> Unable to attach viommu >> -device vfio-pci,host=0000:7d:02.2,bus=pcie.port3,iommufd=iommufd0: vfio >> 0000:7d:02.2: >> Failed to set iommu_device: [iommufd=29] error attach 0000:7d:02.2 (38) >> to id=11: Invalid argument >> >> At present Qemu is not doing any extra validation other than the above >> failure to make sure the user configuration is correct or not. The >> assumption is libvirt will take care of this. >> " >> So in summary, if the libvirt gets it wrong, Qemu will fail with error. >> >> If a more explicit association is required, some help from kernel is required >> to identify the physical SMMUv3 associated with the device. > Again thinking about this, to have an explicit association in the Qemu > command > line between the vSMMUv3 and the phys smmuv3, > > We can possibly add something like, > > -device pxb-pcie,id=pcie.1,bus_nr=8,bus=pcie.0 \ > -device pcie-root-port,id=pcie.port1,bus=pcie.1,chassis=1 \ > -device arm-smmuv3-accel,bus=pcie.1,phys-smmuv3= smmu3.0x0000000100000000 \ > -device vfio-pci,host=0000:7d:02.1,bus=pcie.port1,iommufd=iommufd0 \ > > -device pxb-pcie,id=pcie.2,bus_nr=16,bus=pcie.0 \ > -device pcie-root-port,id=pcie.port2,bus=pcie.2,chassis=2 \ > -device arm-smmuv3-nested,id=smmuv2,pci-bus=pcie.2, phys-smmuv3= > smmu3.0x0000000200000000 \ > -device vfio-pci,host=0000:75:00.1,bus=pcie.port2,iommufd=iommufd0 \ > > etc. > > And Qemu does some checking to make sure that the device is indeed associated > with the specified phys-smmuv3. This can be done going through the sysfs > path checking > which is what I guess libvirt is currently doing to populate the topology. So > basically > Qemu is just replicating that to validate again. > > Or another option is extending the IOMMU_GET_HW_INFO IOCTL to return the phys > smmuv3 base address which can avoid going through the sysfs. > > The only difference between the current approach(kernel failing the attach > implicitly) > and the above is, Qemu can provide a validation of inputs and may be report a > better > error message than just saying " Unable to attach viommu/: Invalid argument". > > If the command line looks Ok, I will go with the sysfs path validation method > first in my > next respin. The command line looks sensible to me. on vfio we use host=6810000.ethernet. Maybe reuse this instead of phys-smmuv3? Thanks Eric > > Please let me know. > > Thanks, > Shameer > > > >
