> -----Original Message----- > From: Peter Maydell <peter.mayd...@linaro.org> > Sent: Thursday, May 8, 2025 2:58 PM > To: Donald Dutile <ddut...@redhat.com> > Cc: Shameerali Kolothum Thodi > <shameerali.kolothum.th...@huawei.com>; Markus Armbruster > <arm...@redhat.com>; Shameer Kolothum via <qemu- > de...@nongnu.org>; qemu-...@nongnu.org; eric.au...@redhat.com; > j...@nvidia.com; nicol...@nvidia.com; berra...@redhat.com; > nath...@nvidia.com; mo...@nvidia.com; smost...@google.com; Linuxarm > <linux...@huawei.com>; Wangzhou (B) <wangzh...@hisilicon.com>; > jiangkunkun <jiangkun...@huawei.com>; Jonathan Cameron > <jonathan.came...@huawei.com>; zhangfei....@linaro.org > Subject: Re: [PATCH v2 1/6] hw/arm/smmuv3: Add support to associate a > PCIe RC > > On Thu, 8 May 2025 at 14:46, Donald Dutile <ddut...@redhat.com> wrote: > > > I would refer to the ARM SMMU spec, Figure 2.3 in the G.a version, where > > it's slightly different; more like: > > > > +------------------+ > > | PCIe Devices | (one device, unless a PCIe switch is btwn the RC & > 'Devices'; > > +------------------+ or, see more typical expansion below) > > | > > +-------------+ > > | PCIe RC A | > > +-------------+ > > | > > +------v---+ +-----------------------------------+ > > | SMMUv3.A | | Wide assortment of other platform | > > | (IOMMU) | | devices not using SMMU | > > +----+-----+ +-----------------------------------+ > > | | | | > > +------+----------------------+---+---+-+ > > | System Interconnect | > > +---------------------------------------+ > > | > > +-------+--------+ +-----+-------------+ > > | System RAM |<--->| CPU (NUMA socket) | > > +----------------+ +-------------------+ > > > > In fact, the PCIe can be quite complex with PCIe bridges, and multiple > Root Ports (RP's), > > and multiple SMMU's: > > > > +--------------+ +--------------+ +--------------+ > > | PCIe Device | | PCIe Device | | PCIe Device | > > +--------------+ +--------------+ +--------------+ > > | | | <--- PCIe bus > > +----------+ +----------+ +----------+ > > | PCIe RP | | PCIe RP | | PCIe RP | <- may be PCI > > Bridge, may > not > > +----------+ +----------+ +----------+ > > | | | > > +----------+ +----------+ +----------+ > > | SMMU | | SMMU | | SMMU | > > +----------+ +----------+ +----------+ > > | | | <- may be a bus, may > > not(hidden from > OS) > > +------------------+------------------+ > > | > > +--------------------------+ > > | PCI RC A | > > +--------------------------+ > > > > > > The final take away: the (QEMU) SMMU/IOMMU must be associated with > a PCIe bus > > OR, the format has to be something like: > > -device smmuv3, id=smmuv3.1 > > -device <blah>, smmu=smmuv3.1 > > where the device <-> SMMU (or if extended to x86, iommu) associativity is > set w/o bus associativity. > > The problem here seems to me to be that in the hardware we're > modelling the SMMU always exists, because it's in the SoC, > but you're trying to arrange for it to be created on the > command line, via -device. > > We don't have any of these problems with the current 'virt' > board code, because we have the board code create the iommu > (if the user asks for it via the iommu machine property), > and it can wire it up to the PCI root complex as needed.
Yes, currently virt creates a SMMUv3 instance and associates it with pcie.0 by default. However, this setup isn't ideal once we introduce support for accelerated SMMUv3, where we need to configure the host SMMUv3 for nested stage translation. This is essential for VFIO-PCI passthrough scenarios, where the guest manages the stage-1 page tables and the host manages stage-2. In such cases, devices may be associated with different host SMMUv3s, and having a single vSMMUv3 in the guest isn't ideal as- -We would need to broadcast TLBIs or perform lookups to identify the corresponding host SMMUv3. -The physical SMMUv3s might differ in functionality. -Some SMMUv3 implementations offer additional virtualization features (e.g., vCMDQ), where the CMDQ is is directly gets mapped to the Guest and invalidations aren't even trapped. Please refer to the earlier RFC [1], which proposed a new arm-smmuv3-accel device. The main feedback on that RFC was to instead convert the existing arm-smmuv3 into a user-creatable device, and introduce acceleration support via an optional property: -device arm-smmuv3,accel=on,.. This series is the first step toward that goal—making the current arm-smmuv3 device user-creatable. Please let me know if there's a better approach to integrating accelerated SMMUv3 support here. Thanks, Shameer [1] https://lore.kernel.org/qemu-devel/20250311141045.66620-1-shameerali.kolothum.th...@huawei.com/
