Hello Dmitry, Slava, On Mon, Oct 11, 2021 at 10:57 AM Dmitry Kozlyuk <dkozl...@oss.nvidia.com> wrote: > > From: Viacheslav Ovsiienko <viachesl...@nvidia.com> > > The primary DPDK process launch might take a long time if initially > allocated memory is large. From practice allocation of 1 TB of memory > over 1 GB hugepages on Linux takes tens of seconds. Fast restart > is highly desired for some applications and launch delay presents > a problem. > > The primary delay happens in this call trace: > rte_eal_init() > rte_eal_memory_init() > rte_eal_hugepage_init() > eal_dynmem_hugepage_init() > eal_memalloc_alloc_seg_bulk() > alloc_seg() > mmap() > > The largest part of the time spent in mmap() is filling the memory > with zeros. Kernel does so to prevent data leakage from a process > that was last using the page. However, in a controlled environment > it may not be the issue, while performance is. (Linux-specific > MAP_UNINITIALIZED flag allows mapping without clearing, but it is > disabled in all popular distributions for the reason above.) > > It is proposed to add a new EAL option: --mem-file FILE1,FILE2,... > to map hugepages "as is" from specified FILEs in hugetlbfs. > Compared to using external memory for the task, EAL option requires > no change to application code, while allowing administrator > to control hugepage sizes and their NUMA affinity. > > Limitations of the feature: > > * Linux-specific (only Linux maps hugepages from files). > * Incompatible with --legacy-mem (partially replaces it). > * Incompatible with --single-file-segments > (--mem-file FILEs can contain as many segments as needed). > * Incompatible with --in-memory (logically). > > A warning about possible security implications is printed > when --mem-file is used. > > Until this patch DPDK allocator always cleared memory on freeing, > so that it did not have to do that on allocation, while new memory > was cleared by the kernel. When --mem-file is in use, DPDK clears memory > after allocation in rte_zmalloc() and does not clean it on freeing. > Effectively user trades fast startup for occasional allocation slowdown > whenever it is absolutely necessary. When memory is recycled, it is > cleared again, which is suboptimal par se, but saves complication > of memory management.
I have some trouble figuring the need for the list of files. Why not use a global knob --mem-clear-on-alloc for this behavior change? > > Signed-off-by: Viacheslav Ovsiienko <viachesl...@nvidia.com> > Signed-off-by: Dmitry Kozlyuk <dkozl...@nvidia.com> -- David Marchand
