On 08-Mar-18 1:36 PM, Pavan Nikhilesh wrote:
Hi Anatoly,
We are currently facing issues with running testpmd on thunderx platform.
The issue seems to be with vfio
EAL: Detected 24 lcore(s)
EAL: Detected 1 NUMA nodes
EAL: No free hugepages reported in hugepages-2048kB
EAL: Multi-process socket /var/run/.rte_unix
EAL: Probing VFIO support...
EAL: VFIO support initialized
EAL: VFIO support not initialized
<snip>
EAL: probe driver: 177d:a053 octeontx_fpavf
EAL: PCI device 0001:01:00.1 on NUMA socket 0
EAL: probe driver: 177d:a034 net_thunderx
EAL: using IOMMU type 1 (Type 1)
EAL: cannot set up DMA remapping, error 22 (Invalid argument)
EAL: 0001:01:00.1 DMA remapping failed, error 22 (Invalid argument)
EAL: Requested device 0001:01:00.1 cannot be used
EAL: PCI device 0001:01:00.2 on NUMA socket 0
<snip>
testpmd: No probed ethernet devices
testpmd: create a new mbuf pool <mbuf_pool_socket_0>: n=251456, size=2176,
socket=0
testpmd: preferred mempool ops selected: ring_mp_mc
EAL: VFIO support not initialized
EAL: VFIO support not initialized
EAL: VFIO support not initialized
Done
This is because rte_service_init() calls rte_calloc() before
rte_bus_probe() and vfio_dma_mem_map fails because iommu type is not set.
Call stack:
gdb) bt
#0 vfio_dma_mem_map (vaddr=281439006359552, iova=11274289152, len=536870912,
do_map=1) at /root/clean/dpdk/lib/librte_eal/linuxapp/eal/eal_vfio.c:967
#1 0x00000000004fd974 in rte_vfio_dma_map (vaddr=281439006359552,
iova=11274289152, len=536870912) at
/root/clean/dpdk/lib/librte_eal/linuxapp/eal/eal_vfio.c:988
#2 0x00000000004fbe78 in vfio_mem_event_callback (type=RTE_MEM_EVENT_ALLOC,
addr=0xfff7a0000000, len=536870912) at
/root/clean/dpdk/lib/librte_eal/linuxapp/eal/eal_vfio.c:240
#3 0x00000000005070ac in eal_memalloc_notify (event=RTE_MEM_EVENT_ALLOC,
start=0xfff7a0000000, len=536870912) at
/root/clean/dpdk/lib/librte_eal/common/eal_common_memalloc.c:177
#4 0x0000000000515c98 in try_expand_heap_primary (heap=0xffffb7fb167c,
pg_sz=536870912, elt_size=8192, socket=0, flags=0, align=128, bound=0,
contig=false) at /root/clean/dpdk/lib/librte_eal/common/malloc_heap.c:247
#5 0x0000000000515e94 in try_expand_heap (heap=0xffffb7fb167c,
pg_sz=536870912, elt_size=8192, socket=0, flags=0, align=128, bound=0,
contig=false) at /root/clean/dpdk/lib/librte_eal/common/malloc_heap.c:327
#6 0x00000000005163a0 in alloc_more_mem_on_socket (heap=0xffffb7fb167c,
size=8192, socket=0, flags=0, align=128, bound=0, contig=false) at
/root/clean/dpdk/lib/librte_eal/common/malloc_heap.c:455
#7 0x0000000000516514 in heap_alloc_on_socket (type=0x85bf90 "rte_services",
size=8192, socket=0, flags=0, align=128, bound=0, contig=false) at
/root/clean/dpdk/lib/librte_eal/common/malloc_heap.c:491
#8 0x0000000000516664 in malloc_heap_alloc (type=0x85bf90 "rte_services",
size=8192, socket_arg=-1, flags=0, align=128, bound=0, contig=false) at
/root/clean/dpdk/lib/librte_eal/common/malloc_heap.c:527
#9 0x0000000000513b54 in rte_malloc_socket (type=0x85bf90 "rte_services",
size=8192, align=128, socket_arg=-1) at
/root/clean/dpdk/lib/librte_eal/common/rte_malloc.c:54
#10 0x0000000000513bc8 in rte_zmalloc_socket (type=0x85bf90 "rte_services",
size=8192, align=128, socket=-1) at /root/clean/dpdk/lib/librte_eal/common/rte_malloc.c:72
#11 0x0000000000513c00 in rte_zmalloc (type=0x85bf90 "rte_services", size=8192,
align=128) at /root/clean/dpdk/lib/librte_eal/common/rte_malloc.c:81
#12 0x0000000000513c90 in rte_calloc (type=0x85bf90 "rte_services", num=64,
size=128, align=128) at /root/clean/dpdk/lib/librte_eal/common/rte_malloc.c:99
#13 0x0000000000518cec in rte_service_init () at
/root/clean/dpdk/lib/librte_eal/common/rte_service.c:81
#14 0x00000000004f55f4 in rte_eal_init (argc=3, argv=0xfffffffff488) at
/root/clean/dpdk/lib/librte_eal/linuxapp/eal/eal.c:959
#15 0x000000000045af5c in main (argc=3, argv=0xfffffffff488) at
/root/clean/dpdk/app/test-pmd/testpmd.c:2483
Also, I have tried running with --legacy-mem but I'm stuck in
`pci_find_max_end_va` loop because `rte_fbarray_find_next_used` always return
0. >
HugePages_Total: 15
HugePages_Free: 11
HugePages_Rsvd: 0
HugePages_Surp: 0
Hugepagesize: 524288 kB
Call Stack:
(gdb) bt
#0 find_next (arr=0xffffb7fb009c, start=0, used=true) at
/root/clean/dpdk/lib/librte_eal/common/eal_common_fbarray.c:248
#1 0x00000000005132a8 in rte_fbarray_find_next_used (arr=0xffffb7fb009c,
start=0) at /root/clean/dpdk/lib/librte_eal/common/eal_common_fbarray.c:700
#2 0x000000000052d030 in pci_find_max_end_va () at
/root/clean/dpdk/drivers/bus/pci/linux/pci.c:138
#3 0x0000000000530ab8 in pci_vfio_map_resource_primary (dev=0xeae700) at
/root/clean/dpdk/drivers/bus/pci/linux/pci_vfio.c:499
#4 0x0000000000530ffc in pci_vfio_map_resource (dev=0xeae700) at
/root/clean/dpdk/drivers/bus/pci/linux/pci_vfio.c:601
#5 0x000000000052ce90 in rte_pci_map_device (dev=0xeae700) at
/root/clean/dpdk/drivers/bus/pci/linux/pci.c:75
#6 0x0000000000531a20 in rte_pci_probe_one_driver (dr=0x997e20
<rte_nicvf_pmd>, dev=0xeae700) at
/root/clean/dpdk/drivers/bus/pci/pci_common.c:164
#7 0x0000000000531c68 in pci_probe_all_drivers (dev=0xeae700) at
/root/clean/dpdk/drivers/bus/pci/pci_common.c:249
#8 0x0000000000531f68 in rte_pci_probe () at
/root/clean/dpdk/drivers/bus/pci/pci_common.c:359
#9 0x000000000050a140 in rte_bus_probe () at
/root/clean/dpdk/lib/librte_eal/common/eal_common_bus.c:98
#10 0x00000000004f55f4 in rte_eal_init (argc=1, argv=0xfffffffff498) at
/root/clean/dpdk/lib/librte_eal/linuxapp/eal/eal.c:967
#11 0x000000000045af5c in main (argc=1, argv=0xfffffffff498) at
/root/clean/dpdk/app/test-pmd/testpmd.c:2483
Am I missing something here?
I'll look into those, thanks!
Btw, i've now set up a github repo with the patchset applied:
https://github.com/anatolyburakov/dpdk
I will be pushing quick fixes there before spinning new revisions, so we
can discover and fix bugs more rapidly. I'll fix compile issues reported
earlier, then i'll take a look at your issues. The latter one seems like
a typo, the former is probably a matter of moving things around a bit.
(also, pull requests welcome if you find it easier to fix things
yourself and submit patches against my tree!)
Thanks for testing.
--
Thanks,
Anatoly
