> > > > > Hi, > > > > > > > > > > I notice a problem with a PCIe endpoint, which is behind a PLX > switch, > > > > assigned to a VM by VFIO. > > > > > The problem is switch report a UR error when the EP tries to DMA > to a > > > > memory zone inside VM's address space. > > > > > Assume that the DMA destination address is between in the VM's ram > > > > > address space, and unfortunately that address value in host's > point of > > > > view just hit the PLX switch upstream port BAR0 memory-mapped IO > range. > > > > > In a result, the DMA will failed because SW think this memory > > > > > request is invalid if the destination address hit its UP's bar. > > > > > Is this a hardware bug or qemu/seabios doesn't maintain a proper > address > > > > space for VM? > > > > > > > > Upstream switch ports are generally single function devices and > > > > therefore governed by 6.12.1.3 (PCIe base spec rev 4.0, v1) which > > > > indicates an ACS capability must not be implemented. We can > > > > therefore read into section > > > > 6.12.2 on interoperability which indicates the interaction between > > > > ACS and non-ACS components, including: > > > > > > > > * When ACS P2P Request Redirect, ACS P2P Completion Redirect, or > both > > > > are being used, certain components in the PCI Express hierarchy > must > > > > support ACS Upstream Forwarding (of Upstream redirected Requests). > > > > Specifically: > > > > ... > > > > Between each ACS component where P2P TLP redirection is enabled > and > > > > its associated Root Port, any intermediate Switches must support > ACS > > > > Upstream Forwarding. Otherwise, how such Switches handle Upstream > > > > redirected TLPs is undefined. > > > > > > > > It's my interpretation therefore that in a configuration where the > > > > switch downstream ports supports ACS, the switch upstream port > > > > must implicitly support upstream forwarding, thus I would consider > this a hardware issue. > > > > The alternative is that we need to poke holes in the VM address > > > > space to account for any possible conflict and assigned device > > > > hot-add becomes nearly a non-starter. Thanks, > > > > > > > > Alex > > > > > > Thanks for your explanation. > > > Do you know are there other vendors' switches that wouldn't result in > this problem? > > > > I've generally thought PLX switches were good, I know the NVIDIA GRID > > cards made use of them and perhaps the multi-GPU Tesla cards still do. > > PLX makes a large variety of switches though, so they could be a > > different family, but also the nature of this issue can mask the > > problem if the address spaces don't overlap in a way that triggers it. > > I should have asked previously, does the PLX switch in question support > ACS on the downstream ports and is it enabled? Thanks, > > Alex
Yes, the PLX switch support ACS on the DPs, and the ACS is enabled. _______________________________________________ vfio-users mailing list vfio-users@redhat.com https://www.redhat.com/mailman/listinfo/vfio-users
