On Mon, May 9, 2016 at 10:06 AM, Jonas Camillus Jeppesen <jona...@sdu.dk> wrote:
> My IOMMU grouping of devices change depending on which pci-e socket I > insert my R9 290 GPU into. For the sake of purchasing a new system I wanted > to discuss the different groupings so I can better choose my new hardware. > > Here are the IOMMU group which contain my GPU with the GPU inserted into > the three different PCIE slots I have (for all groups in the different > configs see attachments). > > #A: GPU in PCIE1 > IOMMU group 1 > 00:01.0 PCI bridge [0604]: Intel Corporation Xeon E3-1200 v3/4th Gen > Core Processor PCI Express x16 Controller [8086:0c01] (rev 06) > 01:00.0 VGA compatible controller [0300]: Advanced Micro Devices, Inc. > [AMD/ATI] Hawaii PRO [Radeon R9 290] [1002:67b1] > 01:00.1 Audio device [0403]: Advanced Micro Devices, Inc. [AMD/ATI] > Hawaii HDMI Audio [1002:aac8] > > #B: GPU in PCIE3 > IOMMU group 1 > 00:01.0 PCI bridge [0604]: Intel Corporation Xeon E3-1200 v3/4th Gen > Core Processor PCI Express x16 Controller [8086:0c01] (rev 06) > 00:01.1 PCI bridge [0604]: Intel Corporation Xeon E3-1200 v3/4th Gen > Core Processor PCI Express x8 Controller [8086:0c05] (rev 06) > 02:00.0 VGA compatible controller [0300]: Advanced Micro Devices, Inc. > [AMD/ATI] Hawaii PRO [Radeon R9 290] [1002:67b1] > 02:00.1 Audio device [0403]: Advanced Micro Devices, Inc. [AMD/ATI] > Hawaii HDMI Audio [1002:aac8] > > #C: GPU in PCIE4 > IOMMU group 14 > 04:00.0 VGA compatible controller [0300]: Advanced Micro Devices, Inc. > [AMD/ATI] Hawaii PRO [Radeon R9 290] [1002:67b1] > 04:00.1 Audio device [0403]: Advanced Micro Devices, Inc. [AMD/ATI] > Hawaii HDMI Audio [1002:aac8] > > I have had success with Arch linux and stock kernel 4.4.1 with config #C > (GPU in PCIE4), but not the others. Is that to be expected without patching > the kernel (i.e. the GPU needs to be in a group all by itself)? > > Can these groups be figured out without plugging the GPU into the > different slots and looking at /sys/kernel/iommu_groups/; deduced from the > chip set specification, inspecting /sys/ in more clever way, or similar? If > yes, then it would be great to collect a list of hardware and its > groupings. That way it would be easier to decide which motherboard and cpu > to get for different setups. > It's very simple, unless you choose a Xeon E5+ or a High End Desktop Processor, then all of the processor based root ports will be grouped together. The difference between #A and #B in your example is simply that the BIOS disables the 00:01.1 root port if there's nothing installed in the slot (it can't disable 00:01.0 in configuration #B because function #0 must be present in order to discover function #1). In example #C you've installed the card in a PCH-based root port for which we have quirks in the kernel to enable isolation. We have quirks for most Intel chipsets to enable this: http://git.kernel.org/cgit/linux/kernel/git/torvalds/linux.git/tree/drivers/pci/quirks.c#n3877 Linux v4.7 should have quirks for Skylake PCH root ports: https://git.kernel.org/cgit/linux/kernel/git/next/linux-next.git/commit/drivers/pci/quirks.c?id=1bf2bf229b64540f91ac6fa3af37c81249037a0b Often the more high-end motherboard manufacturers will have block diagrams of the system layout so you can see which slots are hosted by which device. IIRC supermicro typically does this, not that I'm endorsing their products. All of the above configurations will work, #A and #B simply add restrictions that you can't use the additional CPU-based slots for anything else without (not recommended) using the ACS override patch. For an assigned GPU, it does not matter that the root port itself is included in the group. If however you were trying to make use of a multi-port NIC, with each port assigned to a separate VM, or an SR-IOV device with multiple virtual functions, installing it in such a slot would be a very bad choice for those applications.
_______________________________________________ vfio-users mailing list vfio-users@redhat.com https://www.redhat.com/mailman/listinfo/vfio-users
