On Wed, Feb 13, 2019 at 07:07:11PM +0000, Shahaf Shuler wrote: > Wednesday, February 13, 2019 1:17 PM, Gaëtan Rivet: > > Subject: Re: [PATCH 3/6] bus: introduce DMA memory mapping for external > > memory > > > > On Wed, Feb 13, 2019 at 11:10:23AM +0200, Shahaf Shuler wrote: > > > The DPDK APIs expose 3 different modes to work with memory used for > > DMA: > > > > > > 1. Use the DPDK owned memory (backed by the DPDK provided > > hugepages). > > > This memory is allocated by the DPDK libraries, included in the DPDK > > > memory system (memseg lists) and automatically DMA mapped by the > > DPDK > > > layers. > > > > > > 2. Use memory allocated by the user and register to the DPDK memory > > > systems. This is also referred as external memory. Upon registration > > > of the external memory, the DPDK layers will DMA map it to all needed > > > devices. > > > > > > 3. Use memory allocated by the user and not registered to the DPDK > > > memory system. This is for users who wants to have tight control on > > > this memory. The user will need to explicitly call DMA map function in > > > order to register such memory to the different devices. > > > > > > The scope of the patch focus on #3 above. > > > > > > Currently the only way to map external memory is through VFIO > > > (rte_vfio_dma_map). While VFIO is common, there are other vendors > > > which use different ways to map memory (e.g. Mellanox and NXP). > > > > > > > How are those other vendors' devices mapped initially right now? Are they > > using #2 scheme instead? Then the user will remap everything using #3? > > It is not a re-map, it is a completely different mode for the memory > management. > The first question to ask is "how the application wants to manage its memory" > ? > If it is either #1 or #2 above, no problem to make the mapping internal on > the "other vendor devices" as they can register to the memory event callback > which trigger every time new memory is added to the DPDK memory management > system. > For #3 the memory does not exists in the DPDK memory management system, and > no memory events. Hence the application needs to explicitly call the dma MAP. > The change on this patch is just to make it more generic than calling only > VFIO. >
Right! I mostly used #1 ports and never really thought about other kind of memory management or how they might follow a different logic. Do you think this could be used with a lot of sequential mapping/unmappings happening? I'm thinking for example about a crypto app feeding crypto buffers, being able to directly map the result instead of copying it within buffers might be interesting. But then you'd have to unmap often. - Is the unmap() simple from the app PoV? - Must the mapping remain available for a long time? - Does the app need to call tx_descriptor_status() a few times or does dma_unmap() verify that the mapping is not in use before unmapping? -- Gaëtan Rivet 6WIND
