Currently, memory for device maps is allocated ad-hoc, by calculating end of VA space allocated for hugepages and crossing fingers in hopes that those addresses will be free in primary and secondary processes. This leads to situations such as this:
EAL: Detected 88 lcore(s) EAL: Detected 2 NUMA nodes EAL: Multi-process socket /var/run/dpdk/rte/mp_socket_178323_8af2229603de4 EAL: Probing VFIO support... EAL: VFIO support initialized EAL: PCI device 0000:81:00.0 on NUMA socket 1 EAL: probe driver: 8086:1563 net_ixgbe EAL: Cannot mmap device resource file /sys/bus/pci/devices/0000:81:00.0/resource0 to address: 0x7ff7f5800000 EAL: Requested device 0000:81:00.0 cannot be used EAL: Error - exiting with code: 1 Cause: No Ethernet ports - bye As can be seen from the above log, secondary process has initialized successfully, but device BAR mapping has failed, which resulted in missing ports in the secondary process. This patchset is an attempt to fix this problem once and for all, by using the same method we use for memory to do device mappings as well. That is, by preallocating all of the device memory in advance, so that initialization either succeeds and allows for device mappings, or it fails outright (whereas currently we may be in an in-between kind of situation, where init has succeeded but device mappings have failed). This change breaks the ABI, so it is not for this release. However, i'd like to hear feedback on the approach and whether there are potential problems with other buses/use cases that i didn't think of. Anatoly Burakov (3): fbarray: allow zero-sized elements mem: add device memory reserve/free API bus/pci: use the new device memory API for BAR mapping drivers/bus/pci/linux/pci_init.h | 1 - drivers/bus/pci/linux/pci_uio.c | 11 +- drivers/bus/pci/linux/pci_vfio.c | 27 +- lib/librte_eal/common/eal_common_fbarray.c | 10 +- lib/librte_eal/common/eal_common_memory.c | 270 ++++++++++++++++-- .../common/include/rte_eal_memconfig.h | 18 ++ lib/librte_eal/common/include/rte_memory.h | 40 +++ lib/librte_pci/Makefile | 1 + lib/librte_pci/rte_pci.c | 20 +- 9 files changed, 350 insertions(+), 48 deletions(-) -- 2.17.0
