Hi,
I'm trying to alter an existing program to use the Intel DPDK. I'm
using 1.8.0, compiled by me as a shared library
(CONFIG_RTE_BUILD_COMBINE_LIBS=y and CONFIG_RTE_BUILD_SHARED_LIB=y in
.config) on Ubuntu 12.04. The program needs to allocate large blocks
of memory (between 1 and 4 chunks of 4.5GB, also 1-4 chunks of 2.5
GB). I tried changing my C++ code to use an array allocated using
rte_malloc() instead of the std::vector I was using beforehand, but it
seems the call to rte_malloc() fails. I then made a simple test
program using the DPDK that takes a size to allocate and if that
fails, tries again with sizes of 100MB less, basically the code below.
This is C++ code (well, now that I look it could've been plain C, but
I need C++) compiled with g++-4.6 with '-std=gnu++0x':
int main(int argc, char **argv)
{
int ret = rte_eal_init(argc, argv);
if (ret < 0)
rte_exit(EXIT_FAILURE, "Invalid EAL arguments\n");
argc -= ret;
argv += ret;
[... check argc >= 2]
size_t size = strtoul(argv[1], NULL, 10);
size_t s = size;
for (size_t i = 0; i < 30; ++i) {
printf("Trying to allocate %'zu bytes\n", s);
buf = rte_malloc("test", s, 0);
if (!buf)
printf ("Failed!\n");
else {
printf ("Success!\n");
rte_free(buf);
break;
}
s = s - (100 * 1024ULL * 1024ULL);
}
return 0;
}
I'm getting:
Trying to allocate 4,832,038,656 bytes
Failed!
Trying to allocate 4,727,181,056 bytes
Failed!
[...]
Trying to allocate 2,944,601,856 bytes
Success!
It's not always the same value, but usually somewhere around 3GB
rte_malloc() succeeds. I'm running on a physical (non-VM) NUMA machine
with 2 physical CPUs, each having 64GBs of local memory. The machine
also runs Ubuntu 12.04 server. I've created 16384 hugepages of 2MB:
echo 16384 >
/sys/devices/system/node/node0/hugepages/hugepages-2048kB/nr_hugepages
I'm running the basic app like this:
sudo numactl --membind=0 ~/src/test/dpdk_test/alloc -c 1f -n 4 -w
04:00.0 --socket-mem=16384,0 -- 4832038656
I'm trying to run only on NUMA node 0 and only allocate memory from
there - that's what the app I'm moving to the DPDK works like (using
numactl --membind=x and --cpunodebind=x).
Is there an upper limit on the amount of memory rte_malloc() will try
to allocate? I tried both after a reboot and when the machine had been
running for a while with not much success. Am I missing something?
It's a bit weird to be only able to allocate 3GB out of the 32GB
assigned to the app...
On a related note, what would be a good way to compile the DPDK with
debug info (and preferably -O0)? There's quite a web of .mk files used
and I haven't figured out where the optimization level / debug options
are set.
Thanks in advance,
Stefan.
