Hi Anatoly, First thanks for the patchset, it is a great enhancement.
See question below. Tuesday, September 4, 2018 4:12 PM, Anatoly Burakov: > Subject: [dpdk-dev] [PATCH 00/16] Support externally allocated memory in > DPDK > > This is a proposal to enable using externally allocated memory in DPDK. > > In a nutshell, here is what is being done here: > > - Index internal malloc heaps by NUMA node index, rather than NUMA > node itself (external heaps will have ID's in order of creation) > - Add identifier string to malloc heap, to uniquely identify it > - Each new heap will receive a unique socket ID that will be used by > allocator to decide from which heap (internal or external) to > allocate requested amount of memory > - Allow creating named heaps and add/remove memory to/from those > heaps > - Allocate memseg lists at runtime, to keep track of IOVA addresses > of externally allocated memory > - If IOVA addresses aren't provided, use RTE_BAD_IOVA > - Allow malloc and memzones to allocate from external heaps > - Allow other data structures to allocate from externall heaps > > The responsibility to ensure memory is accessible before using it is on the > shoulders of the user - there is no checking done with regards to validity of > the memory (nor could there be...). That makes sense. However who should be in-charge of mapping this memory for dma access? The user or internally be the PMD when encounter the first packet or while traversing the existing mempools? > > The general approach is to create heap and add memory into it. For any other > process wishing to use the same memory, said memory must first be > attached (otherwise some things will not work). > > A design decision was made to make multiprocess synchronization a manual > process. Due to underlying issues with attaching to fbarrays in secondary > processes, this design was deemed to be better because we don't want to > fail to create external heap in the primary because something in the > secondary has failed when in fact we may not eve have wanted this memory > to be accessible in the secondary in the first place. > > Using external memory in multiprocess is *hard*, because not only memory > space needs to be preallocated, but it also needs to be attached in each > process to allow other processes to access the page table. The attach API call > may or may not succeed, depending on memory layout, for reasons similar to > other multiprocess failures. This is treated as a "known issue" for this > release. > > RFC -> v1 changes: > - Removed the "named heaps" API, allocate using fake socket ID instead > - Added multiprocess support > - Everything is now thread-safe > - Numerous bugfixes and API improvements > > Anatoly Burakov (16): > mem: add length to memseg list > mem: allow memseg lists to be marked as external > malloc: index heaps using heap ID rather than NUMA node > mem: do not check for invalid socket ID > flow_classify: do not check for invalid socket ID > pipeline: do not check for invalid socket ID > sched: do not check for invalid socket ID > malloc: add name to malloc heaps > malloc: add function to query socket ID of named heap > malloc: allow creating malloc heaps > malloc: allow destroying heaps > malloc: allow adding memory to named heaps > malloc: allow removing memory from named heaps > malloc: allow attaching to external memory chunks > malloc: allow detaching from external memory > test: add unit tests for external memory support > > config/common_base | 1 + > config/rte_config.h | 1 + > drivers/bus/fslmc/fslmc_vfio.c | 7 +- > drivers/bus/pci/linux/pci.c | 2 +- > drivers/net/mlx4/mlx4_mr.c | 3 + > drivers/net/mlx5/mlx5.c | 5 +- > drivers/net/mlx5/mlx5_mr.c | 3 + > drivers/net/virtio/virtio_user/vhost_kernel.c | 5 +- > lib/librte_eal/bsdapp/eal/eal.c | 3 + > lib/librte_eal/bsdapp/eal/eal_memory.c | 9 +- > lib/librte_eal/common/eal_common_memory.c | 9 +- > lib/librte_eal/common/eal_common_memzone.c | 8 +- > .../common/include/rte_eal_memconfig.h | 6 +- > lib/librte_eal/common/include/rte_malloc.h | 181 +++++++++ > .../common/include/rte_malloc_heap.h | 3 + > lib/librte_eal/common/include/rte_memory.h | 9 + > lib/librte_eal/common/malloc_heap.c | 287 +++++++++++-- > lib/librte_eal/common/malloc_heap.h | 17 + > lib/librte_eal/common/rte_malloc.c | 383 ++++++++++++++++- > lib/librte_eal/linuxapp/eal/eal.c | 3 + > lib/librte_eal/linuxapp/eal/eal_memalloc.c | 12 +- > lib/librte_eal/linuxapp/eal/eal_memory.c | 4 +- > lib/librte_eal/linuxapp/eal/eal_vfio.c | 17 +- > lib/librte_eal/rte_eal_version.map | 7 + > lib/librte_flow_classify/rte_flow_classify.c | 3 +- > lib/librte_mempool/rte_mempool.c | 31 +- > lib/librte_pipeline/rte_pipeline.c | 3 +- > lib/librte_sched/rte_sched.c | 2 +- > test/test/Makefile | 1 + > test/test/autotest_data.py | 14 +- > test/test/meson.build | 1 + > test/test/test_external_mem.c | 384 ++++++++++++++++++ > test/test/test_malloc.c | 3 + > test/test/test_memzone.c | 3 + > 34 files changed, 1346 insertions(+), 84 deletions(-) create mode 100644 > test/test/test_external_mem.c > > -- > 2.17.1
