On Mon, 14 Sep 2026 09:44:57 -0700 Zhiping Zhang <[email protected]> wrote:
> Thanks Christian -- no worries on the bandwidth, I'll add the Acked-by > to patch 3. > > Hi Alex, Jason, Bjorn, > > Could you help with reviews on patch 4 (vfio/pci) and patch 5 > (RDMA/mlx5), plus acks on patches 1-2 (PCI TPH core) so the set can > travel whole if it goes through the VFIO tree, which is the route > Christian suggests? > > I've also rebased the set onto 7.3 (base 5225b8eec4c9) and retested on > hardware -- PCIe analyzer captures confirming the requested ST and PH > on outbound TLPs, no splats. > > One thing to decide: upstream took 13 for > VFIO_DEVICE_FEATURE_ZPCI_ERROR after v13 went out, so > VFIO_DEVICE_FEATURE_DMA_BUF_TPH becomes 14 in the respin -- let me > know if you'd rather it be something else. > > I'll post v14 after I hear from you. Delta from v13 is the rebase, > that feature-number move, and nothing else functional. IIRC, we're pretty well settled on the vfio-pci front. The feature number does need to be iterated to the next available. We still need acks from PCI and mlx5, and given the extent of the mlx5 changes I expect I need to provide a branch for Jason/Leon once I merge it. Thanks, Alex > On Fri, Sep 11, 2026 at 12:58 AM Christian König > <[email protected]> wrote: > > > > > > > Hi Zhiping, > > > > sorry I'm completely underwater at the moment and don't have time to take > > another look at the full set. > > > > But IIRC you already fixed my documentation requirements and skimming over > > the patch once more I can't see anything wrong of hand. > > > > So feel free to add Acked-by: Christian König <[email protected]> to > > patch "dma-buf: add optional get_pci_tph() callback" and push it upstream > > through the VFIO channels. > > > > Regards, > > Christian. > > > > On 9/10/26 23:46, Zhiping Zhang wrote: > > > Hi Christian, > > > > > > Third ping on this series. The only patch still without review is 3/5, > > > which adds an optional get_pci_tph() callback to dma_buf_ops: > > > > > > https://lore.kernel.org/kvm/[email protected]/ > > > > > > It lets an exporter report the PCIe TPH steering tag for a > > > peer-to-peer mapping. The callback is optional -- exporters that don't > > > implement it behave exactly as today -- and nothing else in dma-buf > > > changes. > > > > > > The rest of the series is settled: Chengwen acked patch 2, the > > > automated review is clean, and there are no open comments from v12. > > > Alex is holding his ack until the dma-buf side has been looked at, so > > > this one callback is gating the whole series. > > > > > > If you have concerns about the API shape or the locking, I'll respin. > > > If someone else should review the dma-buf side, tell me who and I'll > > > take it there. Otherwise I'll check with Alex and Jason next week on > > > how to proceed, since 7.3 is in rc and the next window is about a > > > month out. > > > > > > Thanks, > > > Zhiping > > > > > > On Wed, Sep 2, 2026 at 1:53 PM Zhiping Zhang <[email protected]> wrote: > > >> > > >> Hi Christian, > > >> > > >> Another ping on this series for your attention. Pls see below for more > > >> details. > > >> > > >> Thanks, > > >> Zhiping > > >> > > >> On Fri, Aug 14, 2026 at 11:08 AM Zhiping Zhang <[email protected]> > > >> wrote: > > >>> > > >>> Hi Christian, > > >>> > > >>> A gentle ping on this series, especially patch 3, which adds the > > >>> optional dma-buf get_pci_tph() callback. Could you please review this > > >>> when you have a chance? > > >>> > > >>> https://lore.kernel.org/linux-pci/[email protected]/ > > >>> > > >>> Thanks, > > >>> Zhiping > > >>> > > >>> > > >>> On Fri, Jul 31, 2026 at 2:22 PM Zhiping Zhang <[email protected]> > > >>> wrote: > > >>>> > > >>>> This series adds TLP Processing Hints (TPH) support to the VFIO dma-buf > > >>>> export path, allowing importing drivers (e.g. mlx5) to use the > > >>>> exporter's steering tag when performing peer-to-peer DMA into a > > >>>> VFIO-owned device. > > >>>> > > >>>> There is no separate in-tree vendor kernel driver for the target > > >>>> device: > > >>>> vfio-pci is the in-tree driver and the targeted device is managed > > >>>> from userspace via VFIO passthrough. That is why the ST has to flow > > >>>> through a uAPI: userspace owns the device and its ST table, so it is > > >>>> the > > >>>> entity that can configure a meaningful value for a given dma-buf. The > > >>>> kernel-visible participants are still in-tree: vfio-pci exports the > > >>>> dma-buf and mlx5 imports it. > > >>>> > > >>>> On the effect: the endpoint's PCIe ingress block uses the ST as > > >>>> an in-band instruction for the incoming P2P TLP -- selecting a target > > >>>> cache partition and, on writes, an in-flight operation on the data > > >>>> before it lands. The dma-buf callback keeps this opaque to the > > >>>> framework -- only the producer (userspace owner of the VFIO device) > > >>>> and the consumer (endpoint block) need to interpret the value. The > > >>>> dma-buf get_pci_tph callback itself is optional, but workloads that > > >>>> depend on the endpoint's in-flight operation need it because fallback > > >>>> does not produce the same result. > > >>>> > > >>>> The dma-buf hook is intentionally generic and discoverable rather than > > >>>> a private side channel. The exporter owns the completing address > > >>>> space for the dma-buf and decides whether it can provide a meaningful > > >>>> ST/PH tuple for that completer; the dma-buf core keeps the tuple > > >>>> opaque, > > >>>> and importers merely request the namespace they support and place the > > >>>> returned value on generated TLPs. Exporters that cannot derive a > > >>>> meaningful tuple simply return -EOPNOTSUPP. > > >>>> > > >>>> TPH is advisory: a steering tag that is not honored on the path (for > > >>>> example an intermediate routing element that does not forward the TPH > > >>>> prefix) is ignored and the request completes as an ordinary, > > >>>> non-TPH transaction (PCIe Base 6.4 sec 2.2.7.1). This series therefore > > >>>> targets the same-Root-Port / common-switch topology, where the ST > > >>>> reaches the completer; cross-Root-Port P2P is best-effort and is not > > >>>> gated in the uAPI, since supplying an unused ST is harmless and there > > >>>> is no discoverable "TPH routing" capability to test against. > > >>>> > > >>>> Patch 1 folds the reserved 0b10 "TPH Completer Supported" encoding into > > >>>> "not supported" in get_rp_completer_type(), so only architected values > > >>>> can reach the TPH Requester Enable field. It was previously posted > > >>>> standalone to linux-pci; per Alex Williamson's v12 review it now > > >>>> travels > > >>>> with the series, which removes the cross-tree ordering dependency and > > >>>> lets review tooling apply the series as posted. > > >>>> Patch 2 adds small PCI/TPH type helpers so drivers can query the > > >>>> enabled > > >>>> TPH requester mode and the device's TPH Completer Supported field > > >>>> without reaching into pci_dev internals (and so callers in > > >>>> CONFIG_PCIE_TPH=n builds get a clean fallback). > > >>>> pcie_tph_completer_type() > > >>>> applies the same reserved-encoding fold as get_rp_completer_type(), > > >>>> inlined locally so the helper is self-contained. > > >>>> Patch 3 adds the optional dma_buf_ops::get_pci_tph callback plus the > > >>>> dma_buf_get_pci_tph() importer wrapper so importers can fetch TPH > > >>>> metadata from an exporter under dmabuf->resv. > > >>>> Patch 4 implements get_pci_tph in vfio-pci and adds the new uAPI > > >>>> (VFIO_DEVICE_FEATURE_DMA_BUF_TPH) for userspace to attach the metadata. > > >>>> Patch 5 wires up the mlx5 RDMA driver as a consumer. It also enforces > > >>>> the > > >>>> dma_buf_get_pci_tph() steering-tag lifetime: the tag is only valid for > > >>>> the > > >>>> mapping it was queried against, and the mkey's TPH fields cannot be > > >>>> reprogrammed in place. mlx5 therefore records the registration-time > > >>>> tuple > > >>>> and re-queries after each dma-buf mapping is established under > > >>>> dmabuf->resv; unchanged tuples continue with the existing mkey, while > > >>>> changed or missing tuples fail the remap rather than continue with a > > >>>> stale hint. For vfio-pci BAR dma-bufs this is expected to be a no-op > > >>>> because invalidation is revoke/quiesce, not movement to a new backing > > >>>> placement, and the userspace-provided tuple is not changed by the > > >>>> revoke/un-revoke path. > > >>>> > > >>>> Build-tested with both CONFIG_PCIE_TPH=y and CONFIG_PCIE_TPH=n. > > >>>> Functional validation on the target topology: PCIe analyzer captures > > >>>> on the P2P TLPs confirm the ST emitted by mlx5 matches the value > > >>>> configured through VFIO_DEVICE_FEATURE_DMA_BUF_TPH, and the end-to-end > > >>>> P2P workload only produces results consistent with the endpoint's > > >>>> ST-selected in-flight operation. For example, with userspace > > >>>> configuring 8-bit ST=0xf0 and PH=2, an analyzer capture of a peer-to- > > >>>> peer MWr64 shows "STP MWr64 TC=0 OHC=2 ..." followed by "OHC-B > > >>>> ST=F0h PH=2 HV=1": > > >>>> (TLP Captures) > > >>>> 08000260 -> STP MWr64 TC=0 OHC=2 TS=0 Attr=0 L=8 > > >>>> F0000004 -> RID=4h:0h.0h EP- Tag=F0h > > >>>> E0200000 -> AddrH=000020E0h > > >>>> 00080006 -> AddrL=06000800h > > >>>> 90F00000 -> OHC-B ST=F0h PH=2 HV=1 AMA=0 AV- > > >>>> > > >>>> The dma-buf get_pci_tph interface has also been exercised by a second, > > >>>> independent importer: a different vendor's NIC whose driver is not yet > > >>>> upstream, locally taught to call dma_buf_get_pci_tph(). A PCIe analyzer > > >>>> confirmed the ST it placed on outbound P2P TLPs matches the value > > >>>> configured through VFIO_DEVICE_FEATURE_DMA_BUF_TPH, the same result as > > >>>> with mlx5. Two unrelated importer drivers exercising the callback > > >>>> end-to-end shows the interface is not tied to a single consumer. That > > >>>> importer change is out-of-tree and not part of this series. For that > > >>>> second importer, with userspace configuring 8-bit ST=0xe0 and PH=0, > > >>>> an analyzer capture shows: > > >>>> (TLP Captures) > > >>>> 08200260 -> STP MWr64 TC=0 OHC=2 TS=1 Attr=0 L=8 > > >>>> 4E00004C -> RID=4Ch:0h.0h EP- Tag=4Eh > > >>>> 00170000 -> AddrH=00001700h > > >>>> 00200006 -> AddrL=06002000h > > >>>> 10E00000 -> OHC-B ST=E0h PH=0 HV=1 AMA=0 AV- > > >>>> > > >>>> Changes since v12: > > >>>> Patch 1 (PCI/TPH, new to the series): the reserved-encoding fold, > > >>>> previously posted standalone to linux-pci [1], is now the first patch > > >>>> here (Alex Williamson). Sashiko could not apply v12 because of that > > >>>> external dependency; with the fold in-series and the mlx5 leak fix in > > >>>> linux-next, v13 has none. The code is unchanged from the standalone > > >>>> v3; the Fixes: tag is dropped, since no code path can reach the > > >>>> reserved encoding today and the patch is hardening rather than a fix > > >>>> for observed silicon. > > >>>> > > >>>> Patch 2 (PCI/TPH): inline the reserved-encoding fold in > > >>>> pcie_tph_completer_type() rather than calling the helper that earlier > > >>>> folding revisions added; that helper was dropped in folding v3 per > > >>>> Bjorn Helgaas and Wei Huang. > > >>>> > > >>>> Patch 3 (dma-buf): no functional change. > > >>>> > > >>>> Patch 4 (vfio/pci): also gate the DMA_BUF_TPH feature on > > >>>> vdev->pci_ops->get_dmabuf_phys, matching > > >>>> vfio_pci_core_feature_dma_buf(). Without it PROBE reported the > > >>>> feature > > >>>> as supported on a device that advertises TPH Completer support but > > >>>> cannot export a vfio dma-buf at all, so nothing could ever carry the > > >>>> metadata (Alex Williamson, who raised this to uAPI-affecting > > >>>> severity). > > >>>> > > >>>> Patch 5 (mlx5): keep the !dev->st early-out in mlx5_st_alloc_index() > > >>>> ahead of the pcie_tph_get_cpu_st() call, so splitting out > > >>>> mlx5_st_alloc_index_by_tag() neither adds an ACPI _DSM invocation on > > >>>> devices without ST support nor changes the errno userspace sees when > > >>>> the _DSM lookup fails (Alex Williamson). The commit message now > > >>>> describes this rather than presenting the split as a pure extraction. > > >>>> > > >>>> Previous link: > > >>>> v12: > > >>>> https://lore.kernel.org/linux-pci/[email protected]/ > > >>>> v11: > > >>>> https://lore.kernel.org/linux-pci/[email protected]/ > > >>>> v10: > > >>>> https://lore.kernel.org/linux-pci/[email protected]/ > > >>>> v9: > > >>>> https://lore.kernel.org/dri-devel/[email protected]/ > > >>>> v8: > > >>>> https://lore.kernel.org/dri-devel/[email protected]/ > > >>>> v7: > > >>>> https://lore.kernel.org/dri-devel/[email protected]/ > > >>>> v6: > > >>>> https://lore.kernel.org/dri-devel/[email protected]/ > > >>>> v5: > > >>>> https://lore.kernel.org/dri-devel/[email protected]/ > > >>>> v4: > > >>>> https://lore.kernel.org/linux-pci/[email protected]/ > > >>>> v3: > > >>>> https://lore.kernel.org/linux-pci/[email protected]/ > > >>>> v2: > > >>>> https://lore.kernel.org/linux-pci/[email protected]/ > > >>>> > > >>>> Zhiping Zhang (5): > > >>>> PCI/TPH: treat reserved 0b10 completer encoding as unsupported > > >>>> PCI/TPH: Add requester/completer type helpers > > >>>> dma-buf: add optional get_pci_tph() callback > > >>>> vfio/pci: implement get_pci_tph and DMA_BUF_TPH feature > > >>>> RDMA/mlx5: get tph for p2p access when registering dma-buf mr > > >>>> > > >>>> drivers/dma-buf/dma-buf.c | 32 ++++ > > >>>> drivers/infiniband/hw/mlx5/main.c | 1 + > > >>>> drivers/infiniband/hw/mlx5/mlx5_ib.h | 11 ++ > > >>>> drivers/infiniband/hw/mlx5/mr.c | 151 +++++++++++++++++- > > >>>> drivers/infiniband/hw/mlx5/odp.c | 7 + > > >>>> .../net/ethernet/mellanox/mlx5/core/lib/st.c | 52 +++++- > > >>>> drivers/pci/tph.c | 55 ++++++- > > >>>> drivers/vfio/pci/vfio_pci_core.c | 3 + > > >>>> drivers/vfio/pci/vfio_pci_dmabuf.c | 120 +++++++++++++- > > >>>> drivers/vfio/pci/vfio_pci_priv.h | 13 ++ > > >>>> include/linux/dma-buf.h | 25 +++ > > >>>> include/linux/mlx5/driver.h | 15 ++ > > >>>> include/linux/pci-tph.h | 8 + > > >>>> include/uapi/linux/vfio.h | 43 +++++ > > >>>> 14 files changed, 517 insertions(+), 19 deletions(-) > > >>>> > > >>>> -- > > >>>> 2.53.0-Meta > > >>>> > >
