What happens when the Nvidia GPU driver kernel callback occurs to invalidate the pinned GPU memory region? Doesn't the NIC need to cease all DMA transfers to/from that region before the kernel callback can complete?
From: Elena Agostini <eagost...@nvidia.com> Sent: 04 January 2022 13:55 To: NBU-Contact-Thomas Monjalon (EXTERNAL) <tho...@monjalon.net> Cc: dev@dpdk.org Subject: Re: [PATCH v2] gpudev: pin GPU memory CAUTION: This email originated from outside of the organization. Do not click links or open attachments unless you recognize the sender and know the content is safe. > 04/01/2022 03:41, eagost...@nvidia.com<mailto:eagost...@nvidia.com>: > > From: Elena Agostini <eagost...@nvidia.com<mailto:eagost...@nvidia.com>> > > > > Enable the possibility to make a GPU memory area accessible from > > the CPU. > > > > GPU memory has to be allocated via rte_gpu_mem_alloc(). > > > > This patch allows the gpudev library to pin, through the GPU driver, > > a chunk of GPU memory and to return a memory pointer usable > > by the CPU to access the GPU memory area. > > > > Signed-off-by: Elena Agostini > > <eagost...@nvidia.com<mailto:eagost...@nvidia.com>> > [...] > > +/** > > + * @warning > > + * @b EXPERIMENTAL: this API may change without prior notice. > > + * > > + * Pin a chunk of GPU memory to make it accessible from the CPU > > You should define what means "pin" exactly. > Which properties should we expect? > Thanks for reviewing, this is the kind of discussion I wanted to have. Maybe "pin" is too GDRCopy specific oriented. Here I want to make a GPU memory buffer visible from the CPU. In case of NVIDIA, this means the GPU memory address has to be pinned (virtual address doesn't change) and dma-mapped. Maybe the name should be more like rte_gpu_mem_to_cpu() that's more explicative and generic. > > + * using the memory pointer returned by the function. > > Which function should return the pointer? > rte_gpu_mem_pin is returning an int. Oversight, will fix it. > > > > + * GPU memory has to be allocated via rte_gpu_mem_alloc(). > > Why pinning is not done by rte_gpu_mem_alloc()? > Should it be a flag? rte_gpu_mem_alloc() allocate virtual memory on the GPU that doesn't have to be necessarily shared (pinned) to make it visible from CPU. > > > + * > > + * @param dev_id > > + * Device ID requiring pinned memory. > > + * @param size > > + * Number of bytes to pin. > > + * Requesting 0 will do nothing. > > + * @param ptr > > + * Pointer to the GPU memory area to be pinned. > > + * NULL is a no-op accepted value. > > + > > + * @return > > + * A pointer to the pinned GPU memory usable by the CPU, otherwise NULL > > and rte_errno is set: > > + * - ENODEV if invalid dev_id > > + * - EINVAL if reserved flags > > Which reserved flags? > > > + * - ENOTSUP if operation not supported by the driver > > + * - E2BIG if size is higher than limit > > + * - ENOMEM if out of space > > Is out of space relevant for pinning? Yes, let me add it > > > + * - EPERM if driver error > > + */ > > +__rte_experimental > > +int rte_gpu_mem_pin(int16_t dev_id, size_t size, void *ptr);
