On 02/02/2018 11:56, Peter Xu wrote: > On Thu, Feb 01, 2018 at 07:51:31PM +0000, Dr. David Alan Gilbert wrote: >> * Peter Xu (pet...@redhat.com) wrote: >>> In the past, we prioritized IOMMU migration so that we have such a >>> priority order: >>> >>> IOMMU > PCI Devices >>> >>> When migrating a guest with both vIOMMU and pcie-root-port, we'll always >>> migrate vIOMMU first, since pcie-root-port will be seen to have the same >>> priority of general PCI devices. >>> >>> That's problematic. >>> >>> The thing is that PCI bus number information is stored in the root port, >>> and that is needed by vIOMMU during post_load(), e.g., to figure out >>> context entry for a device. If we don't have correct bus numbers for >>> devices, we won't be able to recover device state of the DMAR memory >>> regions, and things will be messed up. >>> >>> So let's boost the PCIe root ports to be even with higher priority: >>> >>> PCIe Root Port > IOMMU > PCI Devices >>> >>> A smoke test shows that this patch fixes bug 1538953. >> >> Two questions (partially overlapping with what I replied to Michaels): >> a) What happens with multiple IOMMUs? > > If there are more IOMMUs, then the patch will let all the vIOMMUs be > migrated after pcie root ports. > > But a more true answer is that: I don't really know. :) > > Because I even don't know how multiple vIOMMUs will coop with each > other, especially nested.
I am not aware of "nested" IOMMUs. Multiple IOMMUs work together by dividing the bus ranges, when each of them declares in the corresponding ACPI table the bus/device/range is in charge of. However there was a kernel bug some time ago preventing several IOMMUs to work together, I am not sure the problem is solved yet. In nested case, maybe there will be > dependency between vIOMMUs, but I'll avoid thinking about that until > we support more than one vIOMMUs. > >> b) What happens with multiple root ports? > > Same answer as previous one: all of them will be migrated before any > vIOMMUs. > > Note that IMHO we don't care which pcie root port is migrated first - > IMHO they should not depend on each other, but Marcel may correct me. > Right, each Root Port is independent from each other. Thanks, Marcel >> c) How correct is this ordering on different implementations >> (e.g. ARM/Power/etc) > > Currently it won't affect since Intel IOMMU is the only user for > MIG_PRI_IOMMU. After SMMU is merged it may affect (if it uses this > bit), but IMHO it's fine too as long as pcie root ports won't depend > on anything related to SMMU. > > Thanks, >
