On Wed, 26 Jul 2017 09:21:38 +0300 Marcel Apfelbaum <mar...@redhat.com> wrote:
> On 25/07/2017 11:53, 陈博 wrote: > > To accelerate data traversing between devices under the same PCIE Root > > Port or Switch. > > > > See https://lists.nongnu.org/archive/html/qemu-devel/2017-07/msg07209.html > > > > Hi, > > It may be possible, but maybe PCIe Switch assignment is not > the only way to go. > > Adding Alex and Michael for their input on this matter. > More info at: > https://lists.nongnu.org/archive/html/qemu-devel/2017-07/msg07209.html I think you need to look at where the IOMMU is in the topology and what address space the devices are working in when assigned to a VM to realize that it doesn't make any sense to assign switch ports to a VM. GPUs cannot do switch level peer to peer when assigned because they are operating in an I/O virtual address space. This is why we configure ACS on downstream ports to prevent peer to peer. Peer to peer transactions must be forwarded upstream by the switch ports in order to reach the IOMMU for translation. Note however that we do populate peer to peer mappings within the IOMMU, so if the hardware supports it, the IOMMU can reflect the transaction back out to the I/O bus to reach the other device without CPU involvement. Therefore I think the better solution, if it encourages the NVIDIA driver to do the right thing, is to use emulated switches. Assigning the physical switch would really do nothing more than make the PCIe link information more correct in the VM, everything else about the switch would be emulated. Even still, unless you have an I/O topology which integrates the IOMMU into the switch itself, the data flow still needs to go all the way to the root complex to hit the IOMMU before being reflected to the other device. Direct peer to peer between downstream switch ports operates in the wrong address space. Thanks, Alex
