Hi Aaron,
Two things:
1) I am getting an error message at the conclusion of a successful run (see
below). Not sure if this is something I should be looking at or if it is
harmless.
2) I figured out a sequence of steps that can "fix" my broken state
following a successful run. If I do the following, the links are fixed:
a) dpdk-devbind -b i40e 03:00.0 03:00.1 // bind to the normal driver
b) dpdk-devbind -b vfio-pci 03:00.0 03:00.1 // bind back to the vfio-pci
driver
c) physically unplug & plug the XL710 QSFP+ transceiver (mine is optical,
but unplugging just the MTP does not do the trick - I need to unplug the
full transceiver)
Once I complete the sequence above, the link LEDs are back to normal and I
can complete another run of benchmark_rate. This is obviously a bad
solution so if you have any ideas, please let me know.
Rob
[00:00:05.113788990] Testing receive rate 125.000000 Msps on 4 channels
[00:00:05.120454627] Testing transmit rate 125.000000 Msps on 4 channels
[00:00:15.373972384] Benchmark complete.
Benchmark rate summary:
Num received samples: 5099558824
Num dropped samples: 0
Num overruns detected: 0
Num transmitted samples: 4999335588
Num sequence errors (Tx): 0
Num sequence errors (Rx): 0
Num underruns detected: 0
Num late commands: 0
Num timeouts (Tx): 0
Num timeouts (Rx): 0
Done!
i40e_phy_conf_link(): Failed to get PHY capabilities: -7
On Wed, Feb 3, 2021 at 10:16 AM Rob Kossler <rkoss...@nd.edu> wrote:
> Hi Aaron,
> Unfortunately, I already tried playing around with the link timeout
> increasing up to 10 seconds. No luck. But, I am presently troubleshooting
> the issue and trying to switch back and forth between DPDK and normal
> networking. I am finding that normal networking is not working after 1 run
> of DPDK. And, I'm noticing that link LEDs are messed up and normal pings
> are not working. I am playing around with disconnecting / reconnecting
> links in order to get the link LEDs back to normal. My guess is that
> things are not cleaning up as they should.
> Rob
>
> On Wed, Feb 3, 2021 at 9:51 AM Aaron Rossetto via USRP-users <
> usrp-users@lists.ettus.com> wrote:
>
>> I notice in the second and subsequent runs, you get this message from UHD:
>>
>> [ERROR] [DPDK] All DPDK links did not report as up!
>>
>> One of the other issues I've noticed with DPDK (and unfortunately
>> don't have an answer for) is that link detection seems to have issues.
>> I'm not sure if this is an XL710-specific problem or whether it's more
>> widespread, but I added some code to try to mitigate things somewhat
>> in commit eada49e4d. This commit checks the link status at
>> 250-millisecond intervals for up to the link status timeout (default 1
>> second) in case the links take a while to register as up. One thing
>> you could try is overriding the default link status timeout and
>> increasing the value, which you can do by adding a dpdk_link_timeout=X
>> line to the [use_dpdk=1] section of your uhd.conf file, where X is the
>> new timeout in number of milliseconds.
>>
>> Best regards,
>> Aaron
>>
>> On Tue, Feb 2, 2021 at 1:47 PM Rob Kossler <rkoss...@nd.edu> wrote:
>> >
>> > Hi Aaron,
>> > This did indeed help. Now I am able to run ONCE successfully. After
>> that I get an error. Same behavior on both systems. Not yet sure how to
>> clear the error. I played with dpdk_link_timeout and even tried resetting
>> the N310 using "overlay rm n310 && overlay add n310 && systemctl restart
>> usrp-hwd". But no luck.
>> > Rob
>> >
>> > // First run succeeds
>> > root@irisheyes5-hp-z240-sff:~# uhd_image_loader
>> --args="addr=192.168.1.88,type=n3xx,fpga=XG"
>> > [INFO] [UHD] linux; GNU C++ version 9.3.0; Boost_107100;
>> UHD_4.0.0.0-50-ge520e3ff
>> > [INFO] [MPMD] Initializing 1 device(s) in parallel with args:
>> mgmt_addr=192.168.1.88,type=n3xx,product=n310,serial=3144673,claimed=False,skip_init=1
>> > [WARNING] [MPM.RPCServer] A timeout event occured!
>> > [INFO] [MPMD] Claimed device without full initialization.
>> > [INFO] [MPMD IMAGE LOADER] Starting update. This may take a while.
>> > [INFO] [MPM.PeriphManager] Updating component `fpga'
>> > [INFO] [MPM.PeriphManager] Updating component `dts'
>> > [INFO] [MPM.RPCServer] Resetting peripheral manager.
>> > [INFO] [MPM.PeriphManager] Device serial number: 3144673
>> > [INFO] [MPM.PeriphManager] Initialized 2 daughterboard(s).
>> > [INFO] [MPM.PeriphManager] init() called with device args
>> `clock_source=internal,time_source=internal'.
>> > [INFO] [MPMD IMAGE LOADER] Update component function succeeded.
>> > root@irisheyes5-hp-z240-sff:~# benchmark_rate --tx_rate=62.5e6
>> --rx_rate=62.5e6 --channels="0,1,2,3"
>> --args="use_dpdk=1,mgmt_addr=192.168.1.88,addr=192.168.60.2"
>> >
>> > [INFO] [UHD] linux; GNU C++ version 9.3.0; Boost_107100;
>> UHD_4.0.0.0-50-ge520e3ff
>> > EAL: Detected 8 lcore(s)
>> > EAL: Detected 1 NUMA nodes
>> > EAL: Multi-process socket /var/run/dpdk/rte/mp_socket
>> > EAL: No free hugepages reported in hugepages-1048576kB
>> > EAL: Probing VFIO support...
>> > EAL: VFIO support initialized
>> > EAL: PCI device 0000:03:00.0 on NUMA socket -1
>> > EAL: Invalid NUMA socket, default to 0
>> > EAL: probe driver: 8086:1584 net_i40e
>> > EAL: using IOMMU type 1 (Type 1)
>> > EAL: PCI device 0000:03:00.1 on NUMA socket -1
>> > EAL: Invalid NUMA socket, default to 0
>> > EAL: probe driver: 8086:1584 net_i40e
>> > EAL: PCI device 0000:03:00.2 on NUMA socket -1
>> > EAL: Invalid NUMA socket, default to 0
>> > EAL: probe driver: 8086:1584 net_i40e
>> > EAL: PCI device 0000:03:00.3 on NUMA socket -1
>> > EAL: Invalid NUMA socket, default to 0
>> > EAL: probe driver: 8086:1584 net_i40e
>> > [00:00:00.000152] Creating the usrp device with:
>> use_dpdk=1,mgmt_addr=192.168.1.88,addr=192.168.60.2...
>> > [INFO] [MPMD] Initializing 1 device(s) in parallel with args:
>> mgmt_addr=192.168.1.88,type=n3xx,product=n310,serial=3144673,claimed=False,use_dpdk=1,addr=192.168.60.2
>> > [INFO] [MPM.PeriphManager] init() called with device args
>> `mgmt_addr=192.168.1.88,product=n310,use_dpdk=1,clock_source=internal,time_source=internal'.
>> > Using Device: Single USRP:
>> > Device: N300-Series Device
>> > Mboard 0: n310
>> > RX Channel: 0
>> > RX DSP: 0
>> > RX Dboard: A
>> > RX Subdev: Magnesium
>> > RX Channel: 1
>> > RX DSP: 1
>> > RX Dboard: A
>> > RX Subdev: Magnesium
>> > RX Channel: 2
>> > RX DSP: 2
>> > RX Dboard: B
>> > RX Subdev: Magnesium
>> > RX Channel: 3
>> > RX DSP: 3
>> > RX Dboard: B
>> > RX Subdev: Magnesium
>> > TX Channel: 0
>> > TX DSP: 0
>> > TX Dboard: A
>> > TX Subdev: Magnesium
>> > TX Channel: 1
>> > TX DSP: 1
>> > TX Dboard: A
>> > TX Subdev: Magnesium
>> > TX Channel: 2
>> > TX DSP: 2
>> > TX Dboard: B
>> > TX Subdev: Magnesium
>> > TX Channel: 3
>> > TX DSP: 3
>> > TX Dboard: B
>> > TX Subdev: Magnesium
>> >
>> > [00:00:03.21715319] Setting device timestamp to 0...
>> > [INFO] [MULTI_USRP] 1) catch time transition at pps edge
>> > [INFO] [MULTI_USRP] 2) set times next pps (synchronously)
>> > [WARNING] [0/Radio#0] Attempting to set tick rate to 0. Skipping.
>> > [WARNING] [0/Radio#1] Attempting to set tick rate to 0. Skipping.
>> > [WARNING] [0/Radio#1] Attempting to set tick rate to 0. Skipping.
>> > [WARNING] [0/Radio#0] Attempting to set tick rate to 0. Skipping.
>> > Setting TX spp to 1989
>> > [00:00:04.907401082] Testing receive rate 62.500000 Msps on 4 channels
>> > [00:00:04.914615576] Testing transmit rate 62.500000 Msps on 4 channels
>> > [00:00:15.167869894] Benchmark complete.
>> >
>> >
>> > Benchmark rate summary:
>> > Num received samples: 2549794336
>> > Num dropped samples: 0
>> > Num overruns detected: 0
>> > Num transmitted samples: 2499910452
>> > Num sequence errors (Tx): 0
>> > Num sequence errors (Rx): 0
>> > Num underruns detected: 0
>> > Num late commands: 0
>> > Num timeouts (Tx): 0
>> > Num timeouts (Rx): 0
>> >
>> >
>> > Done!
>> >
>> > // Second run fails
>> > root@irisheyes5-hp-z240-sff:~# benchmark_rate --tx_rate=62.5e6
>> --rx_rate=62.5e6 --channels="0,1,2,3"
>> --args="use_dpdk=1,mgmt_addr=192.168.1.88,addr=192.168.60.2"
>> >
>> > [INFO] [UHD] linux; GNU C++ version 9.3.0; Boost_107100;
>> UHD_4.0.0.0-50-ge520e3ff
>> > EAL: Detected 8 lcore(s)
>> > EAL: Detected 1 NUMA nodes
>> > EAL: Multi-process socket /var/run/dpdk/rte/mp_socket
>> > EAL: No free hugepages reported in hugepages-1048576kB
>> > EAL: Probing VFIO support...
>> > EAL: VFIO support initialized
>> > EAL: PCI device 0000:03:00.0 on NUMA socket -1
>> > EAL: Invalid NUMA socket, default to 0
>> > EAL: probe driver: 8086:1584 net_i40e
>> > EAL: using IOMMU type 1 (Type 1)
>> > EAL: PCI device 0000:03:00.1 on NUMA socket -1
>> > EAL: Invalid NUMA socket, default to 0
>> > EAL: probe driver: 8086:1584 net_i40e
>> > EAL: PCI device 0000:03:00.2 on NUMA socket -1
>> > EAL: Invalid NUMA socket, default to 0
>> > EAL: probe driver: 8086:1584 net_i40e
>> > EAL: PCI device 0000:03:00.3 on NUMA socket -1
>> > EAL: Invalid NUMA socket, default to 0
>> > EAL: probe driver: 8086:1584 net_i40e
>> > [ERROR] [DPDK] All DPDK links did not report as up!
>> > EAL: FATAL: already called initialization.
>> > EAL: already called initialization.
>> > [ERROR] [UHD] Device discovery error: RuntimeError: DPDK: All DPDK
>> links did not report as up!
>> > [ERROR] [DPDK] Error with EAL initialization
>> > [ERROR] [X300] X300 Network discovery error RuntimeError: Error with
>> EAL initialization
>> > [00:00:00.000122] Creating the usrp device with:
>> use_dpdk=1,mgmt_addr=192.168.1.88,addr=192.168.60.2...
>> > EAL: FATAL: already called initialization.
>> > EAL: already called initialization.
>> > [ERROR] [DPDK] Error with EAL initialization
>> > [ERROR] [UHD] Device discovery error: RuntimeError: Error with EAL
>> initialization
>> > EAL: FATAL: already called initialization.
>> > EAL: already called initialization.
>> > [ERROR] [DPDK] Error with EAL initialization
>> > [ERROR] [X300] X300 Network discovery error RuntimeError: Error with
>> EAL initialization
>> > Error: LookupError: KeyError: No devices found for ----->
>> > Device Address:
>> > use_dpdk: 1
>> > mgmt_addr: 192.168.1.88
>> > addr: 192.168.60.2
>> >
>> > // Third run fails
>> > root@irisheyes5-hp-z240-sff:~# benchmark_rate --tx_rate=62.5e6
>> --rx_rate=62.5e6 --channels="0,1,2,3"
>> --args="use_dpdk=1,mgmt_addr=192.168.1.88,addr=192.168.60.2"
>> >
>> > [INFO] [UHD] linux; GNU C++ version 9.3.0; Boost_107100;
>> UHD_4.0.0.0-50-ge520e3ff
>> > EAL: Detected 8 lcore(s)
>> > EAL: Detected 1 NUMA nodes
>> > EAL: Multi-process socket /var/run/dpdk/rte/mp_socket
>> > EAL: No free hugepages reported in hugepages-1048576kB
>> > EAL: Probing VFIO support...
>> > EAL: VFIO support initialized
>> > EAL: PCI device 0000:03:00.0 on NUMA socket -1
>> > EAL: Invalid NUMA socket, default to 0
>> > EAL: probe driver: 8086:1584 net_i40e
>> > EAL: using IOMMU type 1 (Type 1)
>> > EAL: PCI device 0000:03:00.1 on NUMA socket -1
>> > EAL: Invalid NUMA socket, default to 0
>> > EAL: probe driver: 8086:1584 net_i40e
>> > EAL: PCI device 0000:03:00.2 on NUMA socket -1
>> > EAL: Invalid NUMA socket, default to 0
>> > EAL: probe driver: 8086:1584 net_i40e
>> > EAL: PCI device 0000:03:00.3 on NUMA socket -1
>> > EAL: Invalid NUMA socket, default to 0
>> > EAL: probe driver: 8086:1584 net_i40e
>> > [ERROR] [DPDK] All DPDK links did not report as up!
>> > EAL: FATAL: already called initialization.
>> > EAL: already called initialization.
>> > [ERROR] [UHD] Device discovery error: RuntimeError: DPDK: All DPDK
>> links did not report as up!
>> > [ERROR] [DPDK] Error with EAL initialization
>> > [ERROR] [X300] X300 Network discovery error RuntimeError: Error with
>> EAL initialization
>> > [00:00:00.000148] Creating the usrp device with:
>> use_dpdk=1,mgmt_addr=192.168.1.88,addr=192.168.60.2...
>> > EAL: FATAL: already called initialization.
>> > EAL: already called initialization.
>> > [ERROR] [DPDK] Error with EAL initialization
>> > [ERROR] [UHD] Device discovery error: RuntimeError: Error with EAL
>> initialization
>> > EAL: FATAL: already called initialization.
>> > EAL: already called initialization.
>> > [ERROR] [DPDK] Error with EAL initialization
>> > [ERROR] [X300] X300 Network discovery error RuntimeError: Error with
>> EAL initialization
>> > Error: LookupError: KeyError: No devices found for ----->
>> > Device Address:
>> > use_dpdk: 1
>> > mgmt_addr: 192.168.1.88
>> > addr: 192.168.60.2
>> >
>> >
>> >
>> > On Tue, Feb 2, 2021 at 11:53 AM Aaron Rossetto via USRP-users <
>> usrp-users@lists.ettus.com> wrote:
>> >>
>> >> On Mon, Feb 1, 2021 at 9:02 PM Rob Kossler via USRP-users
>> >> <usrp-users@lists.ettus.com> wrote:
>> >>
>> >> > Has anyone successfully used DPDK with Ubuntu 20.04, UHD 4.0, Intel
>> XL710 NIC, and N310 (or X310)?
>> >>
>> >> If I remember correctly, I believe DPDK tries to dlopen() *everything*
>> >> in the directory specified by the dpdk_driver parameter in the DPDK
>> >> section of uhd.conf, leading to a lot of errors similar to yours
>> >> ('Invalid ELF header' and the like). Having the correct collection of
>> >> .so files in that directory is key.
>> >>
>> >> What's worked for me in the past when using DPDK with an Intel XL710
>> >> is creating a directory (I used /usr/local/lib/dpdk-pmds) and copying
>> >> a specific set of DPDK .so files into this directory:
>> >> * librte_mempool_ring.so
>> >> * librte_pdump.so (I think this one is optional--I had been trying to
>> >> get packet dumps from DPDK a while back)
>> >> * librte_pmd_i40e.so
>> >> * librte_pmd_ixgbe.so (may be optional?)
>> >> * librte_pmd_pcap.so (this one is also optional, I think)
>> >> * librte_pmd_ring.so
>> >>
>> >> (Symlinking to the actual libraries wherever they get installed
>> >> instead of copying them into the directory would probably work as
>> >> well.)
>> >>
>> >> Then, make sure that the dpdk-driver key in the [use_dpdk=1] section
>> >> of uhd.conf points to that directory:
>> >> dpdk_driver = /usr/local/lib/dpdk-pmds
>> >>
>> >> Hopefully that will resolve the issue and get you a little further
>> >> down the road.
>> >>
>> >> Best regards,
>> >> Aaron
>> >>
>> >> _______________________________________________
>> >> USRP-users mailing list
>> >> USRP-users@lists.ettus.com
>> >> http://lists.ettus.com/mailman/listinfo/usrp-users_lists.ettus.com
>>
>> _______________________________________________
>> USRP-users mailing list
>> USRP-users@lists.ettus.com
>> http://lists.ettus.com/mailman/listinfo/usrp-users_lists.ettus.com
>>
>
_______________________________________________
USRP-users mailing list
USRP-users@lists.ettus.com
http://lists.ettus.com/mailman/listinfo/usrp-users_lists.ettus.com
