Hello David/Dmitry,
After adding -d <path_of_drivers>, I can now see all the pmd shared libs
loading fine and rte_mempool_create works with --no-huge.
Thanks a lot for your support!
Regards,
Mehrdad
-----Original Message-----
From: David Marchand <david.march...@redhat.com>
Sent: December 5, 2024 3:30 AM
To: Alipour, Mehrdad <malip...@ciena.com>
Cc: Dmitry Kozlyuk <dmitry.kozl...@gmail.com>; dev@dpdk.org
Subject: Re: [**EXTERNAL**] Re: rte_mempool_create fails with --no-huge
Hello,
On Thu, Dec 5, 2024 at 12:05 AM Alipour, Mehrdad <malip...@ciena.com> wrote:
>
> Hi Dmitry,
>
> Your observation is good!
> But I have the /usr/lib/dpdk/pmds-24.0 in the LD_LIBRARY_PATH but since the
> host does not have a match glibc, I have to run it with ld-linux-x86-64.so.2
> specifying LD_LIBRARY_PATH with equivalent --library-path.
"the host does not have a match glibc"
Could you clarify?
>
> Here is the entire cmd I use to run testpmd:
>
> sudo /lib/ld-linux-x86-64.so.2 --library-path
> /lib:/usr/lib:/usr/lib/dpdk:/usr/lib/dpdk/pmds-24.0:/ciena/lib
> /usr/bin/dpdk-testpmd -c 000F -n 2 --log-level=eal,8 --no-huge -m 4095
> --no-pci -- -i --nb-cores=2 --total-num-mbufs=2048
>
> Despite the /lib/dpdk/pmds-24.0 libs being visible, the rte_eal_init does not
> load any of these shared libs while it does when I run it inside the VM.
/lib ? or /usr/lib ?
Do you have a symlink between those directories?
Please make sure those paths you listed above do contain DPDK drivers.
> Would appreciate if you can think of any reasons rte_eal_init does not
> trigger loading those shared libs in the host run instance.
About the drivers discovery mechanism: by default, dpdk looks for drivers in
the RTE_EAL_PMD_PATH directory (this path is constructed in config/meson.build).
You can find the value in your build env:
$ grep RTE_EAL_PMD_PATH build-mini/rte_build_config.h #define RTE_EAL_PMD_PATH
"/usr/local/lib64/dpdk/pmds-25.1"
This driver "discovery" is done regardless of the value of LD_LIBRARY_PATH.
But still, (and this could be a bit confusing), loading a driver requires that
the LD_LIBRARY_PATH gives access to other DPDK shared libraries.
A quick check for you, you can try to strace the dpdk process (with strace -f
-e trace=file).
This is a bit verbose, but here is an example with a mini build of mine.
In my case, DPDK is not installed, so I had to set a -d option, but if you
installed DPDK on the host, no need for this option.
$ LD_LIBRARY_PATH=build-mini/lib strace -f -e trace=file
build-mini/app/dpdk-testpmd -c 3 --no-huge -m 40 -d build-mini/drivers -a 0:0.0
--vdev net_null1 --vdev net_null2 --log-level=lib.eal:debug
-- --no-mlockall --total-num-mbufs=2048 -ia ...
EAL: Detected shared linkage of DPDK
newfstatat(AT_FDCWD, "/usr/local/lib64/dpdk/pmds-25.1",
0x7fff01e70590, 0) = -1 ENOENT (No such file or directory) ^^ DPDK tries the
default path, first.
newfstatat(AT_FDCWD, "build-mini/drivers", {st_mode=S_IFDIR|0755, st_size=8192,
...}, 0) = 0 ^^ Then it tries the path I passed via the -d option.
openat(AT_FDCWD, "build-mini/drivers",
O_RDONLY|O_NONBLOCK|O_CLOEXEC|O_DIRECTORY) = 3 newfstatat(3, "",
{st_mode=S_IFDIR|0755, st_size=8192, ...}, AT_EMPTY_PATH) = 0
newfstatat(AT_FDCWD, "build-mini/drivers/librte_bus_pci.so",
{st_mode=S_IFREG|0755, st_size=227744, ...}, 0) = 0 newfstatat(AT_FDCWD,
"build-mini/drivers/librte_bus_vdev.so",
{st_mode=S_IFREG|0755, st_size=62728, ...}, 0) = 0 newfstatat(AT_FDCWD,
"build-mini/drivers/librte_net_vhost.so",
{st_mode=S_IFREG|0755, st_size=217048, ...}, 0) = 0 newfstatat(AT_FDCWD,
"build-mini/drivers/librte_bus_pci.so.25.1",
{st_mode=S_IFREG|0755, st_size=227744, ...}, 0) = 0 newfstatat(AT_FDCWD,
"build-mini/drivers/librte_net_null.so.25.1",
{st_mode=S_IFREG|0755, st_size=378448, ...}, 0) = 0 newfstatat(AT_FDCWD,
"build-mini/drivers/librte_bus_platform.so.25.1",
{st_mode=S_IFREG|0755, st_size=60888, ...}, 0) = 0 newfstatat(AT_FDCWD,
"build-mini/drivers/librte_bus_platform.so",
{st_mode=S_IFREG|0755, st_size=60888, ...}, 0) = 0 newfstatat(AT_FDCWD,
"build-mini/drivers/librte_net_null.so",
{st_mode=S_IFREG|0755, st_size=378448, ...}, 0) = 0 newfstatat(AT_FDCWD,
"build-mini/drivers/librte_net_virtio.so",
{st_mode=S_IFREG|0755, st_size=1268888, ...}, 0) = 0 newfstatat(AT_FDCWD,
"build-mini/drivers/librte_mempool_ring.so",
{st_mode=S_IFREG|0755, st_size=75880, ...}, 0) = 0 newfstatat(AT_FDCWD,
"build-mini/drivers/librte_bus_vdev.so.25.1",
{st_mode=S_IFREG|0755, st_size=62728, ...}, 0) = 0 newfstatat(AT_FDCWD,
"build-mini/drivers/librte_mempool_ring.so.25.1",
{st_mode=S_IFREG|0755, st_size=75880, ...}, 0) = 0 newfstatat(AT_FDCWD,
"build-mini/drivers/librte_net_vhost.so.25.1",
{st_mode=S_IFREG|0755, st_size=217048, ...}, 0) = 0 newfstatat(AT_FDCWD,
"build-mini/drivers/librte_net_virtio.so.25.1",
{st_mode=S_IFREG|0755, st_size=1268888, ...}, 0) = 0 ^^ And it listed all those
drivers accordingly.
...
Now that the driver listing is done, DPDK loads each driver, example with the
first .so:
...
newfstatat(AT_FDCWD, "build-mini/drivers/librte_bus_pci.so",
{st_mode=S_IFREG|0755, st_size=227744, ...}, 0) = 0
EAL: open shared lib build-mini/drivers/librte_bus_pci.so
getcwd("/home/dmarchan/git/pub/dpdk.org/main", 1024) = 37
readlink("/home/dmarchan/git/pub/dpdk.org/main/build-mini",
0x7fff01e70140, 1023) = -1 EINVAL (Invalid argument)
readlink("/home/dmarchan/git/pub/dpdk.org/main/build-mini/drivers",
0x7fff01e70140, 1023) = -1 EINVAL (Invalid argument)
readlink("/home/dmarchan/git/pub/dpdk.org/main/build-mini/drivers/librte_bus_pci.so",
"librte_bus_pci.so.25", 1023) = 20
readlink("/home/dmarchan/git/pub/dpdk.org/main/build-mini/drivers/librte_bus_pci.so.25",
"librte_bus_pci.so.25.1", 1023) = 22
readlink("/home/dmarchan/git/pub/dpdk.org/main/build-mini/drivers/librte_bus_pci.so.25.1",
0x7fff01e70140, 1023) = -1 EINVAL (Invalid argument) newfstatat(AT_FDCWD, "/",
{st_mode=S_IFDIR|0555, st_size=235, ...}, 0) = 0 newfstatat(AT_FDCWD, "/home",
{st_mode=S_IFDIR|0755, st_size=59, ...}, 0) = 0 newfstatat(AT_FDCWD,
"/home/dmarchan", {st_mode=S_IFDIR|0711, st_size=12288, ...}, 0) = 0
newfstatat(AT_FDCWD, "/home/dmarchan/git", {st_mode=S_IFDIR|0775, st_size=27,
...}, 0) = 0 newfstatat(AT_FDCWD, "/home/dmarchan/git/pub",
{st_mode=S_IFDIR|0775, st_size=4096, ...}, 0) = 0 newfstatat(AT_FDCWD,
"/home/dmarchan/git/pub/dpdk.org",
{st_mode=S_IFDIR|0775, st_size=4096, ...}, 0) = 0 newfstatat(AT_FDCWD,
"/home/dmarchan/git/pub/dpdk.org/main",
{st_mode=S_IFDIR|0755, st_size=4096, ...}, 0) = 0 newfstatat(AT_FDCWD,
"/home/dmarchan/git/pub/dpdk.org/main/build-mini",
{st_mode=S_IFDIR|0755, st_size=4096, ...}, 0) = 0 newfstatat(AT_FDCWD,
"/home/dmarchan/git/pub/dpdk.org/main/build-mini/drivers",
{st_mode=S_IFDIR|0755, st_size=8192, ...}, 0) = 0 newfstatat(AT_FDCWD,
"/home/dmarchan/git/pub/dpdk.org/main/build-mini/drivers/librte_bus_pci.so.25.1",
{st_mode=S_IFREG|0755, st_size=227744, ...}, 0) = 0 openat(AT_FDCWD,
"/home/dmarchan/git/pub/dpdk.org/main/build-mini/drivers/librte_bus_pci.so.25.1",
O_RDONLY|O_CLOEXEC) = 3
newfstatat(3, "", {st_mode=S_IFREG|0755, st_size=227744, ...},
AT_EMPTY_PATH) = 0
openat(AT_FDCWD,
"build-mini/lib/glibc-hwcaps/x86-64-v4/librte_pci.so.25",
O_RDONLY|O_CLOEXEC) = -1 ENOENT (No such file or directory) openat(AT_FDCWD,
"build-mini/lib/glibc-hwcaps/x86-64-v3/librte_pci.so.25",
O_RDONLY|O_CLOEXEC) = -1 ENOENT (No such file or directory) openat(AT_FDCWD,
"build-mini/lib/glibc-hwcaps/x86-64-v2/librte_pci.so.25",
O_RDONLY|O_CLOEXEC) = -1 ENOENT (No such file or directory) openat(AT_FDCWD,
"build-mini/lib/librte_pci.so.25", O_RDONLY|O_CLOEXEC) = 3 newfstatat(3, "",
{st_mode=S_IFREG|0755, st_size=23224, ...}, AT_EMPTY_PATH) = 0
getcwd("/home/dmarchan/git/pub/dpdk.org/main", 128) = 37
EAL: Registered [pci] bus.
EAL: bus.pci log level changed from disabled to notice ...
Hope it helps.
--
David Marchand
