Hi Asaf,
On 12-Jul-22 7:05 AM, Asaf Sinai wrote:
Hi,
We run DPDK 19.11.3 with multi processes.
When using external memory for DPDK (instead of huge pages), we are
unable to receive traffic in the secondary processes.
* We have external physical memory regions (excluded from Linux):
/[ULP-NG]# cat /proc/cmdline/
/console=ttyS0,19200 isolcpus=1-127 smp_affinity=1 root=/dev/ram0
rwmode=r net.ifnames=0 biosdevname=0 systemd.show_status=0/
/memmap=0x1000000!0x60000000 *memmap=0x90000000!0x800000000
memmap=0x90000000!0x1800000000*quiet cloud-init=disabled/
* We make all DPDK initializations in the primary process, including
mapping the mentioned memory regions via “/dev/mem”.
After these memory mapping, we register the external memory, as
described in
http://doc.dpdk.org/guides/prog_guide/env_abstraction_layer.html#support-for-externally-allocated-memory
<http://doc.dpdk.org/guides/prog_guide/env_abstraction_layer.html#support-for-externally-allocated-memory>,
and allocate memory pools:
/.../
//* Map physical to virtual *//
/int memFd = open("/dev/mem", O_RDWR);/
/extMemInfo[i].pageSize = pPagesInfo->maxPageSize;/
/extMemInfo[i].memRegSize = 0x0000000080000000;/
/extMemInfo[i].memRegAddr = mmap(NULL, extMemInfo[i].memRegSize,
PROT_READ | PROT_WRITE,/
/ MAP_SHARED, memFd, memRegPhysAddr[i]);/
//
//* Create heap *//
/sprintf(extMemInfo[i].heapName, "extMemHeapSocket_%u", i);/
/rv = rte_malloc_heap_create(extMemInfo[i].heapName);/
//
//* Save heap socket ID *//
/rv = rte_malloc_heap_get_socket(extMemInfo[i].heapName);/
/extMemInfo[i].socketId = rv;/
//
//* Add memory region to heap *//
/rv = rte_malloc_heap_memory_add(extMemInfo[i].heapName,
extMemInfo[i].memRegAddr,/
/ extMemInfo[i].memRegSize, NULL, 0,
extMemInfo[i].pageSize);/
/.../
Please correct me if I'm wrong, but it seems like you're specifying NULL
as IOVA table, so that means external memory will not get IOVA addresses
set. You need to find (probably using rte_mem_virt2phys() function) the
individual page addresses, put them into an array, and pass it as an
argument to add_memory call. See documentation[1], particularly the
iova_addrs[] parameter.
[1]
http://doc.dpdk.org/api/rte__malloc_8h.html#af6360dea35bdf162feeb2b62cf149fd3
//* Allocate memory pool *//
/memPool = rte_mempool_create(poolName, nbufs, DPDK_MBUF_SIZE,
poolCacheSize,/
/ sizeof(struct rte_pktmbuf_pool_private),/
/ rte_pktmbuf_pool_init, NULL,/
/ und_pktmbuf_init, NULL,
extMemInfo[i].socketId, DPDK_NO_FLAGS);/
/.../
Please note, that during calls to “/rte_malloc_heap_memory_add/”, we see
the following warnings:
*/EAL: WARNING! Base virtual address hint (0x2101005000 !=
0x7ffff4627000) not respected!/*
/EAL: This may cause issues with mapping memory into secondary processes/
*/EAL: WARNING! Base virtual address hint (0x210100b000 !=
0x7ffff4619000) not respected!/*
/EAL: This may cause issues with mapping memory into secondary processes/
And after executing “/rte_mempool_create/”, physical addresses of the
allocated memory zones are bad:
Yes, because you did not specify them as part of
rte_malloc_heap_add_memory() call :)
/[Jul 12 12:02:17] [1875] extMemConfig: heapName=extMemHeapSocket_0,
socketId=256, memRegAddr=0x7fff58000000, memRegSize=2415919104,
pageSize=2097152/
/[Jul 12 12:02:18] [1875] memZone: name=MP_ndPool_0, socket_id=256,
vaddr=0x7fffe7e7bec0-0x7fffe7ffffc0,
*paddr=0xffffffffffffffff-0x1840ff*, len=1589504, hugepage_sz=2MB/
* In the next step, we spawn the secondary processes from the primary
one, using fork().
This way, all DPDK data and memory mappings are the same on all
processes. For example:
This is a recipe for disaster, because DPDK creates internal data
structures that are mapped as files, and primary process expects to be
the sole owner and user of those data structures. If you would like to
create a secondary process using fork() at all, you should fork() first,
and then call rte_eal_init() in the secondary process. A forked primary
process is not what we mean when we say "secondary process".
In addition, see documentation[1] for `rte_malloc_heap_memory_add()` -
there's a note there that states:
Before accessing this memory in other processes, it needs to be attached
in each of those processes by calling rte_malloc_heap_memory_attach in
each other process.
So, not only you need to call secondary process EAL init, you will also
have to call `rte_malloc_heap_memory_attach()` before you can use your
shared external memory in your secondary process.
Hope this is helpful!
_Mapping in primary process:_
/cat /proc/1875/maps/
/.../
*/7ffec8000000-7fff58000000 rw-s 1800000000 00:06
5 /dev/mem/*
*/7fff58000000-7fffe8000000 rw-s 800000000 00:06
5 /dev/mem/*
/.../
_Mapping in secondary process:_
/cat /proc/2676/maps/
/.../
*/7ffec8000000-7fff58000000 rw-s 1800000000 00:06
5 /dev/mem/*
*/7fff58000000-7fffe8000000 rw-s 800000000 00:06
5 /dev/mem/*
/.../
* Unfortunately, when traffic is received by the secondary processes,
we see the following printout on the primary process:
*/i40e_dev_alarm_handler(): ICR0: malicious programming detected/*
* Additionally, we saw no example of using external physical shared
memory on secondary processes.
DPDK test applications show usage of anonymous private memory on the
primary process only.
What are we missing?
Can you please advise?
Thanks,
Asaf
Radware
--
Thanks,
Anatoly
