I notice in the second and subsequent runs, you get this message from UHD:
[ERROR] [DPDK] All DPDK links did not report as up!
One of the other issues I've noticed with DPDK (and unfortunately
don't have an answer for) is that link detection seems to have issues.
I'm not sure if this is an XL710-specific problem or whether it's more
widespread, but I added some code to try to mitigate things somewhat
in commit eada49e4d. This commit checks the link status at
250-millisecond intervals for up to the link status timeout (default 1
second) in case the links take a while to register as up. One thing
you could try is overriding the default link status timeout and
increasing the value, which you can do by adding a dpdk_link_timeout=X
line to the [use_dpdk=1] section of your uhd.conf file, where X is the
new timeout in number of milliseconds.
Best regards,
Aaron
On Tue, Feb 2, 2021 at 1:47 PM Rob Kossler <rkoss...@nd.edu> wrote:
>
> Hi Aaron,
> This did indeed help. Now I am able to run ONCE successfully. After that I
> get an error. Same behavior on both systems. Not yet sure how to clear the
> error. I played with dpdk_link_timeout and even tried resetting the N310
> using "overlay rm n310 && overlay add n310 && systemctl restart usrp-hwd".
> But no luck.
> Rob
>
> // First run succeeds
> root@irisheyes5-hp-z240-sff:~# uhd_image_loader
> --args="addr=192.168.1.88,type=n3xx,fpga=XG"
> [INFO] [UHD] linux; GNU C++ version 9.3.0; Boost_107100;
> UHD_4.0.0.0-50-ge520e3ff
> [INFO] [MPMD] Initializing 1 device(s) in parallel with args:
> mgmt_addr=192.168.1.88,type=n3xx,product=n310,serial=3144673,claimed=False,skip_init=1
> [WARNING] [MPM.RPCServer] A timeout event occured!
> [INFO] [MPMD] Claimed device without full initialization.
> [INFO] [MPMD IMAGE LOADER] Starting update. This may take a while.
> [INFO] [MPM.PeriphManager] Updating component `fpga'
> [INFO] [MPM.PeriphManager] Updating component `dts'
> [INFO] [MPM.RPCServer] Resetting peripheral manager.
> [INFO] [MPM.PeriphManager] Device serial number: 3144673
> [INFO] [MPM.PeriphManager] Initialized 2 daughterboard(s).
> [INFO] [MPM.PeriphManager] init() called with device args
> `clock_source=internal,time_source=internal'.
> [INFO] [MPMD IMAGE LOADER] Update component function succeeded.
> root@irisheyes5-hp-z240-sff:~# benchmark_rate --tx_rate=62.5e6
> --rx_rate=62.5e6 --channels="0,1,2,3"
> --args="use_dpdk=1,mgmt_addr=192.168.1.88,addr=192.168.60.2"
>
> [INFO] [UHD] linux; GNU C++ version 9.3.0; Boost_107100;
> UHD_4.0.0.0-50-ge520e3ff
> EAL: Detected 8 lcore(s)
> EAL: Detected 1 NUMA nodes
> EAL: Multi-process socket /var/run/dpdk/rte/mp_socket
> EAL: No free hugepages reported in hugepages-1048576kB
> EAL: Probing VFIO support...
> EAL: VFIO support initialized
> EAL: PCI device 0000:03:00.0 on NUMA socket -1
> EAL: Invalid NUMA socket, default to 0
> EAL: probe driver: 8086:1584 net_i40e
> EAL: using IOMMU type 1 (Type 1)
> EAL: PCI device 0000:03:00.1 on NUMA socket -1
> EAL: Invalid NUMA socket, default to 0
> EAL: probe driver: 8086:1584 net_i40e
> EAL: PCI device 0000:03:00.2 on NUMA socket -1
> EAL: Invalid NUMA socket, default to 0
> EAL: probe driver: 8086:1584 net_i40e
> EAL: PCI device 0000:03:00.3 on NUMA socket -1
> EAL: Invalid NUMA socket, default to 0
> EAL: probe driver: 8086:1584 net_i40e
> [00:00:00.000152] Creating the usrp device with:
> use_dpdk=1,mgmt_addr=192.168.1.88,addr=192.168.60.2...
> [INFO] [MPMD] Initializing 1 device(s) in parallel with args:
> mgmt_addr=192.168.1.88,type=n3xx,product=n310,serial=3144673,claimed=False,use_dpdk=1,addr=192.168.60.2
> [INFO] [MPM.PeriphManager] init() called with device args
> `mgmt_addr=192.168.1.88,product=n310,use_dpdk=1,clock_source=internal,time_source=internal'.
> Using Device: Single USRP:
> Device: N300-Series Device
> Mboard 0: n310
> RX Channel: 0
> RX DSP: 0
> RX Dboard: A
> RX Subdev: Magnesium
> RX Channel: 1
> RX DSP: 1
> RX Dboard: A
> RX Subdev: Magnesium
> RX Channel: 2
> RX DSP: 2
> RX Dboard: B
> RX Subdev: Magnesium
> RX Channel: 3
> RX DSP: 3
> RX Dboard: B
> RX Subdev: Magnesium
> TX Channel: 0
> TX DSP: 0
> TX Dboard: A
> TX Subdev: Magnesium
> TX Channel: 1
> TX DSP: 1
> TX Dboard: A
> TX Subdev: Magnesium
> TX Channel: 2
> TX DSP: 2
> TX Dboard: B
> TX Subdev: Magnesium
> TX Channel: 3
> TX DSP: 3
> TX Dboard: B
> TX Subdev: Magnesium
>
> [00:00:03.21715319] Setting device timestamp to 0...
> [INFO] [MULTI_USRP] 1) catch time transition at pps edge
> [INFO] [MULTI_USRP] 2) set times next pps (synchronously)
> [WARNING] [0/Radio#0] Attempting to set tick rate to 0. Skipping.
> [WARNING] [0/Radio#1] Attempting to set tick rate to 0. Skipping.
> [WARNING] [0/Radio#1] Attempting to set tick rate to 0. Skipping.
> [WARNING] [0/Radio#0] Attempting to set tick rate to 0. Skipping.
> Setting TX spp to 1989
> [00:00:04.907401082] Testing receive rate 62.500000 Msps on 4 channels
> [00:00:04.914615576] Testing transmit rate 62.500000 Msps on 4 channels
> [00:00:15.167869894] Benchmark complete.
>
>
> Benchmark rate summary:
> Num received samples: 2549794336
> Num dropped samples: 0
> Num overruns detected: 0
> Num transmitted samples: 2499910452
> Num sequence errors (Tx): 0
> Num sequence errors (Rx): 0
> Num underruns detected: 0
> Num late commands: 0
> Num timeouts (Tx): 0
> Num timeouts (Rx): 0
>
>
> Done!
>
> // Second run fails
> root@irisheyes5-hp-z240-sff:~# benchmark_rate --tx_rate=62.5e6
> --rx_rate=62.5e6 --channels="0,1,2,3"
> --args="use_dpdk=1,mgmt_addr=192.168.1.88,addr=192.168.60.2"
>
> [INFO] [UHD] linux; GNU C++ version 9.3.0; Boost_107100;
> UHD_4.0.0.0-50-ge520e3ff
> EAL: Detected 8 lcore(s)
> EAL: Detected 1 NUMA nodes
> EAL: Multi-process socket /var/run/dpdk/rte/mp_socket
> EAL: No free hugepages reported in hugepages-1048576kB
> EAL: Probing VFIO support...
> EAL: VFIO support initialized
> EAL: PCI device 0000:03:00.0 on NUMA socket -1
> EAL: Invalid NUMA socket, default to 0
> EAL: probe driver: 8086:1584 net_i40e
> EAL: using IOMMU type 1 (Type 1)
> EAL: PCI device 0000:03:00.1 on NUMA socket -1
> EAL: Invalid NUMA socket, default to 0
> EAL: probe driver: 8086:1584 net_i40e
> EAL: PCI device 0000:03:00.2 on NUMA socket -1
> EAL: Invalid NUMA socket, default to 0
> EAL: probe driver: 8086:1584 net_i40e
> EAL: PCI device 0000:03:00.3 on NUMA socket -1
> EAL: Invalid NUMA socket, default to 0
> EAL: probe driver: 8086:1584 net_i40e
> [ERROR] [DPDK] All DPDK links did not report as up!
> EAL: FATAL: already called initialization.
> EAL: already called initialization.
> [ERROR] [UHD] Device discovery error: RuntimeError: DPDK: All DPDK links did
> not report as up!
> [ERROR] [DPDK] Error with EAL initialization
> [ERROR] [X300] X300 Network discovery error RuntimeError: Error with EAL
> initialization
> [00:00:00.000122] Creating the usrp device with:
> use_dpdk=1,mgmt_addr=192.168.1.88,addr=192.168.60.2...
> EAL: FATAL: already called initialization.
> EAL: already called initialization.
> [ERROR] [DPDK] Error with EAL initialization
> [ERROR] [UHD] Device discovery error: RuntimeError: Error with EAL
> initialization
> EAL: FATAL: already called initialization.
> EAL: already called initialization.
> [ERROR] [DPDK] Error with EAL initialization
> [ERROR] [X300] X300 Network discovery error RuntimeError: Error with EAL
> initialization
> Error: LookupError: KeyError: No devices found for ----->
> Device Address:
> use_dpdk: 1
> mgmt_addr: 192.168.1.88
> addr: 192.168.60.2
>
> // Third run fails
> root@irisheyes5-hp-z240-sff:~# benchmark_rate --tx_rate=62.5e6
> --rx_rate=62.5e6 --channels="0,1,2,3"
> --args="use_dpdk=1,mgmt_addr=192.168.1.88,addr=192.168.60.2"
>
> [INFO] [UHD] linux; GNU C++ version 9.3.0; Boost_107100;
> UHD_4.0.0.0-50-ge520e3ff
> EAL: Detected 8 lcore(s)
> EAL: Detected 1 NUMA nodes
> EAL: Multi-process socket /var/run/dpdk/rte/mp_socket
> EAL: No free hugepages reported in hugepages-1048576kB
> EAL: Probing VFIO support...
> EAL: VFIO support initialized
> EAL: PCI device 0000:03:00.0 on NUMA socket -1
> EAL: Invalid NUMA socket, default to 0
> EAL: probe driver: 8086:1584 net_i40e
> EAL: using IOMMU type 1 (Type 1)
> EAL: PCI device 0000:03:00.1 on NUMA socket -1
> EAL: Invalid NUMA socket, default to 0
> EAL: probe driver: 8086:1584 net_i40e
> EAL: PCI device 0000:03:00.2 on NUMA socket -1
> EAL: Invalid NUMA socket, default to 0
> EAL: probe driver: 8086:1584 net_i40e
> EAL: PCI device 0000:03:00.3 on NUMA socket -1
> EAL: Invalid NUMA socket, default to 0
> EAL: probe driver: 8086:1584 net_i40e
> [ERROR] [DPDK] All DPDK links did not report as up!
> EAL: FATAL: already called initialization.
> EAL: already called initialization.
> [ERROR] [UHD] Device discovery error: RuntimeError: DPDK: All DPDK links did
> not report as up!
> [ERROR] [DPDK] Error with EAL initialization
> [ERROR] [X300] X300 Network discovery error RuntimeError: Error with EAL
> initialization
> [00:00:00.000148] Creating the usrp device with:
> use_dpdk=1,mgmt_addr=192.168.1.88,addr=192.168.60.2...
> EAL: FATAL: already called initialization.
> EAL: already called initialization.
> [ERROR] [DPDK] Error with EAL initialization
> [ERROR] [UHD] Device discovery error: RuntimeError: Error with EAL
> initialization
> EAL: FATAL: already called initialization.
> EAL: already called initialization.
> [ERROR] [DPDK] Error with EAL initialization
> [ERROR] [X300] X300 Network discovery error RuntimeError: Error with EAL
> initialization
> Error: LookupError: KeyError: No devices found for ----->
> Device Address:
> use_dpdk: 1
> mgmt_addr: 192.168.1.88
> addr: 192.168.60.2
>
>
>
> On Tue, Feb 2, 2021 at 11:53 AM Aaron Rossetto via USRP-users
> <usrp-users@lists.ettus.com> wrote:
>>
>> On Mon, Feb 1, 2021 at 9:02 PM Rob Kossler via USRP-users
>> <usrp-users@lists.ettus.com> wrote:
>>
>> > Has anyone successfully used DPDK with Ubuntu 20.04, UHD 4.0, Intel XL710
>> > NIC, and N310 (or X310)?
>>
>> If I remember correctly, I believe DPDK tries to dlopen() *everything*
>> in the directory specified by the dpdk_driver parameter in the DPDK
>> section of uhd.conf, leading to a lot of errors similar to yours
>> ('Invalid ELF header' and the like). Having the correct collection of
>> .so files in that directory is key.
>>
>> What's worked for me in the past when using DPDK with an Intel XL710
>> is creating a directory (I used /usr/local/lib/dpdk-pmds) and copying
>> a specific set of DPDK .so files into this directory:
>> * librte_mempool_ring.so
>> * librte_pdump.so (I think this one is optional--I had been trying to
>> get packet dumps from DPDK a while back)
>> * librte_pmd_i40e.so
>> * librte_pmd_ixgbe.so (may be optional?)
>> * librte_pmd_pcap.so (this one is also optional, I think)
>> * librte_pmd_ring.so
>>
>> (Symlinking to the actual libraries wherever they get installed
>> instead of copying them into the directory would probably work as
>> well.)
>>
>> Then, make sure that the dpdk-driver key in the [use_dpdk=1] section
>> of uhd.conf points to that directory:
>> dpdk_driver = /usr/local/lib/dpdk-pmds
>>
>> Hopefully that will resolve the issue and get you a little further
>> down the road.
>>
>> Best regards,
>> Aaron
>>
>> _______________________________________________
>> USRP-users mailing list
>> USRP-users@lists.ettus.com
>> http://lists.ettus.com/mailman/listinfo/usrp-users_lists.ettus.com
_______________________________________________
USRP-users mailing list
USRP-users@lists.ettus.com
http://lists.ettus.com/mailman/listinfo/usrp-users_lists.ettus.com
