That's an interesting combination. This system is old enough that you have something more like a traditional North Bridge, South Bridge setup. Intel moved the memory controller to the processor in Nehalem & Westmere, but your high speed Root Ports are on the IOH, part of chipset, not the processor. These are root ports 00:01.0, 00:02.0, 00:03.0, and 00:07.0. These root ports do have ACS support, so things below them are isolated from one another. Rather than a PCH, you have an IOH. Those root ports are 00:1c.0, 00:1c.2, and 00:1c.4. Those do not support ACS, do not have quirks, and are part of a multifunction device, so they and everything behind them is grouped together, that's your IOMMU group 9.
Now, back to your IOH paths, devices 15:00.* are grouped together because they lack ACS between the functions (I believe that Marvell SATA controller is one of the ones that uses the wrong requester ID for DMA too). 14:00.0 is in it's own group. Then come the PCIe switches. The downstream ports on the switches do not support ACS, so everything downstream of them will be grouped together. This is why group 13 contains not only the GTX 770, but also the downstream port at 10:00.0. That's fine for a GPU assignment use case, but would not work well for things like multi-port NICs or SR-IOV devices where you want to assign each function to a different VM. The only other interesting one is 08:00.0, where you've got a second layer of PCIe switches. Everything in that hierarchy is already grouped together due to the lack of ACS at 06:01.0, making group 21 pretty large. (I think that other Marvell SATA controller also has DMA issues, but it should already have quirks in the kernel) So it looks ok to me. The isolation isn't perfect, but it's ok for GPU assignment.
_______________________________________________ vfio-users mailing list vfio-users@redhat.com https://www.redhat.com/mailman/listinfo/vfio-users
