Hello!
I want to get DPDK working with UHD 4.6 on our USRP X410. This might be a long
post, but I want to make sure I provide as much information as possible. Below
is a diagram of relevant network connections:
```
*---------------------* *---------------------*
```
```
| | NIC QSFP Port 0 | | QSFP28 Port 1 | |
```
```
| | ens1f0np0 <>---------------<> sfp1 | |
```
```
| | 192.168.20.1 | | 192.168.20.2 | |
```
```
| |-----------------| |-----------------| |
```
```
| H | NIC QSFP Port 1 | | QSFP28 Port 0 | X |
```
```
| O | ens1f1np1 <>---------------<> sfp0 | 4 |
```
```
| S | 192.168.10.1 | | 192.168.10.2 | 1 |
```
```
| T |-----------------| *-----------* |-----------------| 0 |
```
```
| | Ethernet Port 1 | |Router | | Ethernet Port | |
```
```
| | eno8403 <>-|192.168.1.1|-<> eth0 | |
```
```
| | 192.168.1.20 | | | | 192.168.1.19 | |
```
```
*---------------------* *-----------* *---------------------*
```
The Host computer has the following specs:
OS: Ubuntu 20.04.6 LTS
CPU: Intel(R) Xeon(R) Silver 4310
RAM: 64 GB
NIC: Mellanox Technologies MT2892 Family \[ConnectX-6 Dx\]
UHD: UHD_4.6.0.HEAD-0-g50fa3baa
DPDK: DPDK_19.11
I am using a uhd.conf with:
```
[use_dpdk=1]
```
```
dpdk_mtu=9000
```
```
dpdk_corelist=0,1,2,3
```
```
dpdk_num_mbufs=8192
```
```
dpdk_mbuf_cache_size=64
```
```
[dpdk_mac=MAC_ADDRESS]
```
```
dpdk_lcore = 1
```
```
dpdk_ipv4 = 192.168.20.1/24
```
```
[dpdk_mac=MAC_ADDRESS]
```
```
dpdk_lcore = 2
```
```
dpdk_ipv4 = 192.168.10.1/24
```
When I run “dpdk-devbind.py -s“, I get:
```
Network devices using kernel driver
```
```
===================================
```
```
0000:04:00.0 'NetXtreme BCM5720 2-port Gigabit Ethernet PCIe 165f' if=eno8303
drv=tg3 unused=vfio-pci *Active*
```
```
0000:04:00.1 'NetXtreme BCM5720 2-port Gigabit Ethernet PCIe 165f' if=eno8403
drv=tg3 unused=vfio-pci *Active*
```
```
0000:31:00.0 'MT2892 Family [ConnectX-6 Dx] 101d' if=ens1f0np0 drv=mlx5_core
unused=vfio-pci *Active*
```
```
0000:31:00.1 'MT2892 Family [ConnectX-6 Dx] 101d' if=ens1f1np1 drv=mlx5_core
unused=vfio-pci *Active*
```
And the output of uhd_find_devices is:
```
[INFO] [UHD] linux; GNU C++ version 9.4.0; Boost_107100; DPDK_19.11;
UHD_4.6.0.HEAD-0-g50fa3baa
```
```
--------------------------------------------------
```
```
-- UHD Device 0
```
```
--------------------------------------------------
```
```
Device Address:
```
```
serial: 326E872
```
```
addr: 192.168.20.2
```
```
claimed: False
```
```
fpga: CG_400
```
```
mgmt_addr: 192.168.1.19
```
```
mgmt_addr: 192.168.10.2
```
```
mgmt_addr: 192.168.20.2
```
```
name: ni-x4xx-326E872
```
```
product: x410
```
```
type: x4xx
```
This all makes me believe I at least have DPDK set up correctly. I followed the
instructions here
(https://files.ettus.com/manual/page_dpdk.html#dpdk_nic_config) for GRUB.
When I try to run any UHD-based script with DPDK, like ‘ uhd_find_devices
--args "use_dpdk=1" ‘, I get errors like:
```
EAL: Couldn't get fd on hugepage file
```
```
EAL: Couldn't get fd on hugepage file
```
```
EAL: error allocating rte services array
```
```
EAL: FATAL: rte_service_init() failed
```
```
EAL: rte_service_init() failed
```
```
[ERROR] [DPDK] Error with EAL initialization
```
And the device is not found. After some searching, I found that the reason is
DPDK doesn’t play nicely without elevated permissions. Going forward, assume I
run everything as the root user. Running the same command as before then yields:
```
[INFO] [UHD] linux; GNU C++ version 9.4.0; Boost_107100; DPDK_19.11;
UHD_4.6.0.HEAD-0-g50fa3baa
```
```
EAL: Detected 24 lcore(s)
```
```
EAL: Detected 2 NUMA nodes
```
```
EAL: Multi-process socket /var/run/dpdk/rte/mp_socket
```
```
EAL: Selected IOVA mode 'VA'
```
```
EAL: No available hugepages reported in hugepages-1048576kB
```
```
EAL: Probing VFIO support...
```
```
EAL: VFIO support initialized
```
```
EAL: PCI device 0000:31:00.0 on NUMA socket 0
```
```
EAL: probe driver: 15b3:101d net_mlx5
```
```
EAL: PCI device 0000:31:00.1 on NUMA socket 0
```
```
EAL: probe driver: 15b3:101d net_mlx5
```
```
[ERROR] [DPDK] Could not find route to destination address 10.149.11.255
```
```
[ERROR] [X300] X300 Network discovery error RuntimeError: DPDK: Could not find
route to destination address 10.149.11.255
```
```
[ERROR] [DPDK] Could not find route to destination address 192.168.1.255
```
```
[ERROR] [X300] X300 Network discovery error RuntimeError: DPDK: Could not find
route to destination address 192.168.1.255
```
```
[ERROR] [DPDK] Could not find route to destination address 172.17.255.255
```
```
[ERROR] [X300] X300 Network discovery error RuntimeError: DPDK: Could not find
route to destination address 172.17.255.255
```
```
--------------------------------------------------
```
```
-- UHD Device 0
```
```
--------------------------------------------------
```
```
Device Address:
```
```
serial: 326E872
```
```
addr: 192.168.1.19
```
```
claimed: False
```
```
fpga: CG_400
```
```
mgmt_addr: 192.168.1.19
```
```
name: ni-x4xx-326E872
```
```
product: x410
```
```
type: x4xx
```
Now the device is found, but with errors. When I further specify all the
addresses with:
```
uhd_find_devices --args
“addr=192.168.10.2,second_addr=192.168.20.2,mgmt_addr=192.168.1.19,use_dpdk=1"
```
The errors are gone.
When I try running an example with DPDK though, like:
```
./benchmark_rate --rx_rate 491.52e6 --args
"addr=192.168.10.2,second_addr=192.168.20.2,mgmt_addr=192.168.1.19,use_dpdk=1"
--duration 1
```
I get:
```
[INFO] [UHD] linux; GNU C++ version 9.4.0; Boost_107100; DPDK_19.11;
UHD_4.6.0.HEAD-0-g50fa3baa
```
```
EAL: Detected 24 lcore(s)
```
```
EAL: Detected 2 NUMA nodes
```
```
EAL: Multi-process socket /var/run/dpdk/rte/mp_socket
```
```
EAL: Selected IOVA mode 'VA'
```
```
EAL: No available hugepages reported in hugepages-1048576kB
```
```
EAL: Probing VFIO support...
```
```
EAL: VFIO support initialized
```
```
EAL: PCI device 0000:31:00.0 on NUMA socket 0
```
```
EAL: probe driver: 15b3:101d net_mlx5
```
```
EAL: PCI device 0000:31:00.1 on NUMA socket 0
```
```
EAL: probe driver: 15b3:101d net_mlx5
```
```
[00:00:00.001242] Creating the usrp device with:
addr=192.168.10.2,second_addr=192.168.20.2,mgmt_addr=192.168.1.19,use_dpdk=1...
```
```
[INFO] [MPMD] Initializing 1 device(s) in parallel with args:
mgmt_addr=192.168.1.19,type=x4xx,product=x410,serial=326E872,name=ni-x4xx-326E872,fpga=CG_400,claimed=False,addr=192.168.10.2,second_addr=192.168.20.2,use_dpdk=1
```
```
[INFO] [MPM.PeriphManager] init() called with device args
`fpga=CG_400,mgmt_addr=192.168.1.19,name=ni-x4xx-326E872,product=x410,second_addr=192.168.20.2,use_dpdk=1,clock_source=internal,time_source=internal,initializing=True'.
```
```
[ERROR] [RFNOC::MGMT] EnvironmentError: IOError: Timed out getting recv buff
for management transaction
```
```
[ERROR] [RFNOC::GRAPH] IO Error during GSM initialization. EnvironmentError:
IOError: Timed out getting recv buff for management transaction
```
```
[ERROR] [RFNOC::GRAPH] Caught exception while initializing graph:
EnvironmentError: IOError: Timed out getting recv buff for management
transaction
```
```
Error: RuntimeError: Failure to create rfnoc_graph.
```
What is the issue? How do I fix things? When I first set up DPDK, I also set up
HugePages, although I don't exactly remember how. As I only have a basic
understanding of Linux, I'm not sure I set that up correctly, but running:
```
cat /proc/meminfo | grep Huge
```
results in:
```
AnonHugePages: 0 kB
```
```
ShmemHugePages: 0 kB
```
```
FileHugePages: 0 kB
```
```
HugePages_Total: 2048
```
```
HugePages_Free: 2045
```
```
HugePages_Rsvd: 0
```
```
HugePages_Surp: 0
```
```
Hugepagesize: 2048 kB
```
```
Hugetlb: 4194304 kB
```
_______________________________________________
USRP-users mailing list -- usrp-users@lists.ettus.com
To unsubscribe send an email to usrp-users-le...@lists.ettus.com
