On Thu, 13 Sep 2018 20:55:39 -0600 Alex Williamson <alex.william...@redhat.com> wrote:
> On Fri, 14 Sep 2018 01:25:20 +0000 > "Wuzongyong (Euler Dept)" <cordius...@huawei.com> wrote: > > > > > Hi, > > > > > > > > I notice a problem with a PCIe endpoint, which is behind a PLX switch, > > > > > > > assigned to a VM by VFIO. > > > > The problem is switch report a UR error when the EP tries to DMA to a > > > > > > > memory zone inside VM's address space. > > > > Assume that the DMA destination address is between in the VM's ram > > > > address space, and unfortunately that address value in host's point of > > > > > > > view just hit the PLX switch upstream port BAR0 memory-mapped IO range. > > > > > > > In a result, the DMA will failed because SW think this memory request > > > > is invalid if the destination address hit its UP's bar. > > > > Is this a hardware bug or qemu/seabios doesn't maintain a proper > > > > address > > > space for VM? > > > > > > Upstream switch ports are generally single function devices and therefore > > > governed by 6.12.1.3 (PCIe base spec rev 4.0, v1) which indicates an ACS > > > capability must not be implemented. We can therefore read into section > > > 6.12.2 on interoperability which indicates the interaction between ACS and > > > non-ACS components, including: > > > > > > * When ACS P2P Request Redirect, ACS P2P Completion Redirect, or both > > > are being used, certain components in the PCI Express hierarchy must > > > support ACS Upstream Forwarding (of Upstream redirected Requests). > > > Specifically: > > > ... > > > Between each ACS component where P2P TLP redirection is enabled and > > > its associated Root Port, any intermediate Switches must support ACS > > > Upstream Forwarding. Otherwise, how such Switches handle Upstream > > > redirected TLPs is undefined. > > > > > > It's my interpretation therefore that in a configuration where the switch > > > downstream ports supports ACS, the switch upstream port must implicitly > > > support upstream forwarding, thus I would consider this a hardware issue. > > > The alternative is that we need to poke holes in the VM address space to > > > account for any possible conflict and assigned device hot-add becomes > > > nearly a non-starter. Thanks, > > > > > > Alex > > > > Thanks for your explanation. > > Do you know are there other vendors' switches that wouldn't result in this > > problem? > > I've generally thought PLX switches were good, I know the NVIDIA GRID > cards made use of them and perhaps the multi-GPU Tesla cards still do. > PLX makes a large variety of switches though, so they could be a > different family, but also the nature of this issue can mask the > problem if the address spaces don't overlap in a way that triggers it. I should have asked previously, does the PLX switch in question support ACS on the downstream ports and is it enabled? Thanks, Alex _______________________________________________ vfio-users mailing list vfio-users@redhat.com https://www.redhat.com/mailman/listinfo/vfio-users
