On 3/10/2020 2:55 PM, David Marchand wrote: > On Tue, Mar 10, 2020 at 3:49 PM Ferruh Yigit <ferruh.yi...@intel.com> wrote: >> >> On 3/6/2020 2:48 PM, David Marchand wrote: >>> Since 18.05 and the memory subsystem rework, EAL reserves some big >>> (unused) mappings. >>> >>> In testpmd, we have been locking all pages to avoid page faults during >>> benchmark/performance regression tests [1]. >>> However, asking for locking all the pages triggers issues on FreeBSD [2] >>> and becomes really heavy in some Linux configurations (see [3], [4]). >>> >>> This patch changes the behavior so that testpmd only lock pages >>> containing .text by default. >>> >>> 1: https://git.dpdk.org/dpdk/commit/?id=1c036b16c284 >>> 2: https://git.dpdk.org/dpdk/commit/?id=fb7b8b32cd95 >>> 3: https://bugzilla.redhat.com/show_bug.cgi?id=1786923 >>> 4: http://mails.dpdk.org/archives/dev/2020-February/158477.html >>> >>> Signed-off-by: David Marchand <david.march...@redhat.com> >> >> <...> >> >>> @@ -3455,6 +3456,42 @@ signal_handler(int signum) >>> } >>> } >>> >>> +static void >>> +lock_pages(const void *_addr, size_t _len, const char *prefix) >>> +{ >>> + const void *addr; >>> + size_t pagesize; >>> + size_t len; >>> + >>> + /* While Linux does not care, FreeBSD mlock expects page aligned >>> + * address (according to the man). >>> + */ >>> + pagesize = sysconf(_SC_PAGESIZE); >>> + addr = RTE_PTR_ALIGN_FLOOR(_addr, pagesize); >>> + len = _len + ((uintptr_t)_addr & (pagesize - 1)); >>> + if (mlock(addr, len)) { >>> + TESTPMD_LOG(NOTICE, "%s: mlock %p (0x%zx) aligned to %p >>> (0x%zx) failed with error \"%s\"\n", >>> + prefix, _addr, _len, addr, len, strerror(errno)); >>> + } >>> +} >>> + >>> +static int >>> +lock_text_cb(struct dl_phdr_info *info, __rte_unused size_t size, >>> + __rte_unused void *data) >>> +{ >>> + int i; >>> + >>> + for (i = 0; i < info->dlpi_phnum; i++) { >>> + void *addr; >>> + >>> + if (info->dlpi_phdr[i].p_memsz == 0) >>> + continue; >>> + addr = (void *)(info->dlpi_addr + info->dlpi_phdr[i].p_vaddr); >>> + lock_pages(addr, info->dlpi_phdr[i].p_memsz, info->dlpi_name); >>> + } >>> + return 0; >>> +} >> >> +1 to the idea, testpmd initialization was taking too lock without >> '--no-mlockall', but this code looks complex for the application level. >> >> We can do this for testpmd but does all applications need to do something >> similar? If so can we have a solution on eal level instead? > > I submitted a patch on the EAL side. > This makes mlockall way lighter, since it skips pages marked with PROT_NONE. > http://patchwork.dpdk.org/patch/66469/ >
Cool, With that patch timing improves a lot, in my system testpmd initialization reduced from 38s to 9s. (it was 6s with --no-mlockall). Do we still need this testpmd patch?
