RE: [RFC PATCH v3 06/15] hw/arm/smmuv3-accel: Restrict accelerated SMMUv3 to vfio-pci endpoints with iommufd

[Shameerali Kolothum Thodi via]

> -----Original Message-----
> From: Nicolin Chen <nicol...@nvidia.com>
> Sent: Tuesday, July 15, 2025 6:59 PM
> To: Duan, Zhenzhong <zhenzhong.d...@intel.com>
> Cc: Shameerali Kolothum Thodi
> <shameerali.kolothum.th...@huawei.com>; qemu-...@nongnu.org;
> qemu-devel@nongnu.org; eric.au...@redhat.com;
> peter.mayd...@linaro.org; j...@nvidia.com; ddut...@redhat.com;
> berra...@redhat.com; nath...@nvidia.com; mo...@nvidia.com;
> smost...@google.com; Linuxarm <linux...@huawei.com>; Wangzhou (B)
> <wangzh...@hisilicon.com>; jiangkunkun <jiangkun...@huawei.com>;
> Jonathan Cameron <jonathan.came...@huawei.com>;
> zhangfei....@linaro.org; shameerkolot...@gmail.com
> Subject: Re: [RFC PATCH v3 06/15] hw/arm/smmuv3-accel: Restrict
> accelerated SMMUv3 to vfio-pci endpoints with iommufd

...

> > >+    if (pdev && !smmuv3_accel_pdev_allowed(pdev, &vfio_pci)) {
> > >+        error_report("Device(%s) not allowed. Only PCIe root complex
> > >devices "
> > >+                     "or PCI bridge devices or vfio-pci endpoint devices
> > >with "
> > >+                     "iommufd as backend is allowed with
> > >arm-smmuv3,accel=on",
> > >+                     pdev->name);
> > >+        exit(1);
> >
> > Seems aggressive for a hotplug, could we fail hotplug instead of kill
> QEMU?

That's right. I will try to see whether it is possible to do a dev->hotplugged
check here.
 
> Hotplug will unlikely be supported well, as it would introduce
> too much complication.
> 
> With iommufd, a vIOMMU object is allocated per device (vfio). If
> the device fd (cdev) is not yet given to the QEMU. It isn't able
> to allocate a vIOMMU object when creating a VM.
> 
> While a vIOMMU object can be allocated at a later stage once the
> device is hotplugged. But things like IORT mappings aren't able
> to get refreshed since the OS is likely already booted.

Why do we need IORT mappings to be refreshed during hotplug?
AFAICS, the mappings are created per host bridge Ids. And how is this
different from a host machine doing hotplug?

 Even an
> IOMMU capability sync via the hw_info ioctl will be difficult to
> do at the runtime post the guest iommu driver's initialization.

We had some discussion on this "at least one vfio-pci" restriction
for accelerated mode previously here.
https://lore.kernel.org/qemu-devel/z6ttclq35ui12...@redhat.com/#t

I am not sure we reached any consensus on that. The 3 different approaches
discussed were,

1. The current one used here. At least one cold plugged vfio-pci device
   so that  we can retrieve the host SMMUV3 HW_INFO as per current
  IOMMUFD APIs.

2. A new IOMMUFD API to retrieve HW_INFO without a device. 

3. A fully specified vSMMUv3 through Qemu command line so that we
   don't need HW_INFO from kernel.

We're going with option one for now, but completely blocking hotplug
because of it  feels a bit too restrictive to me.

The real issue (for now), as I see it, is that we need some way to remember
the Guest SMMUv3 <-> Host SMMUv3 mapping after the guest has booted.
That way, even if all devices tied to a Guest SMMUv3 get hot-unplugged,
QEMU can still block attaching a device that belongs to a different Host
SMMUv3.

Thanks,
Shameer

> I am not 100% sure. But I think Intel model could have a similar
> problem if the guest boots with zero cold-plugged device and then
> hot-plugs a PASID-capable device at the runtime, when the guest-
> level IOMMU driver is already inited?
> 
> FWIW, Shameer's cover-letter has the following line:
>  "At least one vfio-pci device must currently be cold-plugged to
>   a PCIe root complex associated with arm-smmuv3,accel=on."
> 
> Perhaps there should be a similar highlight in this smmuv3-accel
> file as well (@Shameer).
> 
> Nicolin