On 20/07/2025 13:30, Leon Romanovsky wrote:
External email: Use caution opening links or attachments
On Fri, Jul 18, 2025 at 02:51:07PM +0300, Yonatan Maman wrote:
From: Yonatan Maman <yma...@nvidia.com>
This patch series aims to enable Peer-to-Peer (P2P) DMA access in
GPU-centric applications that utilize RDMA and private device pages. This
enhancement reduces data transfer overhead by allowing the GPU to directly
expose device private page data to devices such as NICs, eliminating the
need to traverse system RAM, which is the native method for exposing
device private page data.
To fully support Peer-to-Peer for device private pages, the following
changes are proposed:
`Memory Management (MM)`
* Leverage struct pagemap_ops to support P2P page operations: This
modification ensures that the GPU can directly map device private pages
for P2P DMA.
* Utilize hmm_range_fault to support P2P connections for device private
pages (instead of Page fault)
`IB Drivers`
Add TRY_P2P_REQ flag for the hmm_range_fault call: This flag indicates the
need for P2P mapping, enabling IB drivers to efficiently handle P2P DMA
requests.
`Nouveau driver`
Add support for the Nouveau p2p_page callback function: This update
integrates P2P DMA support into the Nouveau driver, allowing it to handle
P2P page operations seamlessly.
`MLX5 Driver`
Utilize NIC Address Translation Service (ATS) for ODP memory, to optimize
DMA P2P for private device pages. Also, when P2P DMA mapping fails due to
inaccessible bridges, the system falls back to standard DMA, which uses host
memory, for the affected PFNs
I'm probably missing something very important, but why can't you always
perform p2p if two devices support it? It is strange that IB and not HMM
has a fallback mode.
Thanks
P2P mapping can fail even when both devices support it, due to PCIe
bridge limitations or IOMMU restrictions that block direct P2P access.
The fallback is in IB rather than HMM because HMM only manages memory
pages - it doesn't do DMA mapping. The IB driver does the actual DMA
operations, so it knows when P2P mapping fails and can fall back to
copying through system memory.
In fact, hmm_range_fault doesn't have information about the destination
device that will perform the DMA mapping.
Previous version:
https://lore.kernel.org/linux-mm/20241201103659.420677-1-yma...@nvidia.com/
https://lore.kernel.org/linux-mm/20241015152348.3055360-1-yma...@nvidia.com/
Yonatan Maman (5):
mm/hmm: HMM API to enable P2P DMA for device private pages
nouveau/dmem: HMM P2P DMA for private dev pages
IB/core: P2P DMA for device private pages
RDMA/mlx5: Enable P2P DMA with fallback mechanism
RDMA/mlx5: Enabling ATS for ODP memory
drivers/gpu/drm/nouveau/nouveau_dmem.c | 110 +++++++++++++++++++++++++
drivers/infiniband/core/umem_odp.c | 4 +
drivers/infiniband/hw/mlx5/mlx5_ib.h | 6 +-
drivers/infiniband/hw/mlx5/odp.c | 24 +++++-
include/linux/hmm.h | 3 +-
include/linux/memremap.h | 8 ++
mm/hmm.c | 57 ++++++++++---
7 files changed, 195 insertions(+), 17 deletions(-)
--
2.34.1
