[USRP-users] X410 - FPGA unresponsive after several RFNoC Graph Creations

Maximilian Matthé Fri, 04 Nov 2022 01:21:06 -0700

Dear all,

I have a problem with our USRP X410 and RFNoC. During development of an RfNoc 
application, one often restarts the test program, which creates an RfNoc graph 
and performs some connections on the device. However, after around 10-15 
program starts, the firmware/FPGA/driver yields errors that it's not reachable 
anymore. Below is a MWE yielding the error:


Test skript:
$ cat graph_error.py
import uhd
import time

if __name__ == '__main__':
    time.sleep(1)
    uhd.rfnoc.RfnocGraph("addr=localhost")
    time.sleep(1)
    print("Exiting...")

which I run in a loop directly on the USRP X410, wtih the output attached below.

$ for i in `seq 20`; do echo $i && python3 graph_error.py ; done

After the error occurs, the only way to get it back to working is to reboot the 
USRP or do `systemctl restart usrp-hwd`, which interrupts the dev workflow.

My question is you can reproduce this? Is this a bug? Can I work around the 
crash? If it's a bug, I should file it to the UHD github repository, right?

Thank you,
Maximilian Matthe

$ for i in `seq 20`; do echo $i && python3 graph_error.py ; done
1
[INFO] [UHD] linux; GNU C++ version 9.2.0; Boost_107100; UHD_4.2.0.0-0-g46a70d85
[INFO] [MPMD] Initializing 1 device(s) in parallel with args: 
mgmt_addr=127.0.0.1,type=x4xx,product=x410,serial=323F754,name=NE-LAB-X
410-01,fpga=X4_400,claimed=False,addr=localhost
[INFO] [MPM.PeriphManager] init() called with device args 
`fpga=X4_400,mgmt_addr=127.0.0.1,name=NE-LAB-X410-01,product=x410,clock_sou
rce=internal,time_source=internal'.
Exiting...
2
[INFO] [UHD] linux; GNU C++ version 9.2.0; Boost_107100; UHD_4.2.0.0-0-g46a70d85
[INFO] [MPMD] Initializing 1 device(s) in parallel with args: 
mgmt_addr=127.0.0.1,type=x4xx,product=x410,serial=323F754,name=NE-LAB-X
410-01,fpga=X4_400,claimed=False,addr=localhost
[INFO] [MPM.PeriphManager] init() called with device args 
`fpga=X4_400,mgmt_addr=127.0.0.1,name=NE-LAB-X410-01,product=x410,clock_sou
rce=internal,time_source=internal'.
Exiting...
3
[INFO] [UHD] linux; GNU C++ version 9.2.0; Boost_107100; UHD_4.2.0.0-0-g46a70d85
[INFO] [MPMD] Initializing 1 device(s) in parallel with args: 
mgmt_addr=127.0.0.1,type=x4xx,product=x410,serial=323F754,name=NE-LAB-X
410-01,fpga=X4_400,claimed=False,addr=localhost
[INFO] [MPM.PeriphManager] init() called with device args 
`fpga=X4_400,mgmt_addr=127.0.0.1,name=NE-LAB-X410-01,product=x410,clock_sou
rce=internal,time_source=internal'.
Exiting...
4
[INFO] [UHD] linux; GNU C++ version 9.2.0; Boost_107100; UHD_4.2.0.0-0-g46a70d85
[INFO] [MPMD] Initializing 1 device(s) in parallel with args: 
mgmt_addr=127.0.0.1,type=x4xx,product=x410,serial=323F754,name=NE-LAB-X
410-01,fpga=X4_400,claimed=False,addr=localhost
[INFO] [MPM.PeriphManager] init() called with device args 
`fpga=X4_400,mgmt_addr=127.0.0.1,name=NE-LAB-X410-01,product=x410,clock_sou
rce=internal,time_source=internal'.
Exiting...
5
[INFO] [UHD] linux; GNU C++ version 9.2.0; Boost_107100; UHD_4.2.0.0-0-g46a70d85
[INFO] [MPMD] Initializing 1 device(s) in parallel with args: 
mgmt_addr=127.0.0.1,type=x4xx,product=x410,serial=323F754,name=NE-LAB-X
410-01,fpga=X4_400,claimed=False,addr=localhost
[INFO] [MPM.PeriphManager] init() called with device args 
`fpga=X4_400,mgmt_addr=127.0.0.1,name=NE-LAB-X410-01,product=x410,clock_sou
rce=internal,time_source=internal'.
Exiting...
6
[INFO] [UHD] linux; GNU C++ version 9.2.0; Boost_107100; UHD_4.2.0.0-0-g46a70d85
[INFO] [MPMD] Initializing 1 device(s) in parallel with args: 
mgmt_addr=127.0.0.1,type=x4xx,product=x410,serial=323F754,name=NE-LAB-X
410-01,fpga=X4_400,claimed=False,addr=localhost
[INFO] [MPM.PeriphManager] init() called with device args 
`fpga=X4_400,mgmt_addr=127.0.0.1,name=NE-LAB-X410-01,product=x410,clock_sou
rce=internal,time_source=internal'.
Exiting...
7
[INFO] [UHD] linux; GNU C++ version 9.2.0; Boost_107100; UHD_4.2.0.0-0-g46a70d85
[INFO] [MPMD] Initializing 1 device(s) in parallel with args: 
mgmt_addr=127.0.0.1,type=x4xx,product=x410,serial=323F754,name=NE-LAB-X
410-01,fpga=X4_400,claimed=False,addr=localhost
[INFO] [MPM.PeriphManager] init() called with device args 
`fpga=X4_400,mgmt_addr=127.0.0.1,name=NE-LAB-X410-01,product=x410,clock_sou
rce=internal,time_source=internal'.
Exiting...
8
[INFO] [UHD] linux; GNU C++ version 9.2.0; Boost_107100; UHD_4.2.0.0-0-g46a70d85
[INFO] [MPMD] Initializing 1 device(s) in parallel with args: 
mgmt_addr=127.0.0.1,type=x4xx,product=x410,serial=323F754,name=NE-LAB-X
410-01,fpga=X4_400,claimed=False,addr=localhost
[INFO] [MPM.PeriphManager] init() called with device args 
`fpga=X4_400,mgmt_addr=127.0.0.1,name=NE-LAB-X410-01,product=x410,clock_sou
rce=internal,time_source=internal'.
Exiting...
9
[INFO] [UHD] linux; GNU C++ version 9.2.0; Boost_107100; UHD_4.2.0.0-0-g46a70d85
[INFO] [MPMD] Initializing 1 device(s) in parallel with args: 
mgmt_addr=127.0.0.1,type=x4xx,product=x410,serial=323F754,name=NE-LAB-X
410-01,fpga=X4_400,claimed=False,addr=localhost
[INFO] [MPM.PeriphManager] init() called with device args 
`fpga=X4_400,mgmt_addr=127.0.0.1,name=NE-LAB-X410-01,product=x410,clock_sou
rce=internal,time_source=internal'.
Exiting...
10
[INFO] [UHD] linux; GNU C++ version 9.2.0; Boost_107100; UHD_4.2.0.0-0-g46a70d85
[INFO] [MPMD] Initializing 1 device(s) in parallel with args: 
mgmt_addr=127.0.0.1,type=x4xx,product=x410,serial=323F754,name=NE-LAB-X
410-01,fpga=X4_400,claimed=False,addr=localhost
[INFO] [MPM.PeriphManager] init() called with device args 
`fpga=X4_400,mgmt_addr=127.0.0.1,name=NE-LAB-X410-01,product=x410,clock_sou
rce=internal,time_source=internal'.
Exiting...
11
[INFO] [UHD] linux; GNU C++ version 9.2.0; Boost_107100; UHD_4.2.0.0-0-g46a70d85
[INFO] [MPMD] Initializing 1 device(s) in parallel with args: 
mgmt_addr=127.0.0.1,type=x4xx,product=x410,serial=323F754,name=NE-LAB-X
410-01,fpga=X4_400,claimed=False,addr=localhost
[INFO] [MPM.PeriphManager] init() called with device args 
`fpga=X4_400,mgmt_addr=127.0.0.1,name=NE-LAB-X410-01,product=x410,clock_sou
rce=internal,time_source=internal'.
[ERROR] [RFNOC::GRAPH] Caught exception while initializing graph: RfnocError: 
Specified destination address is unreachable
Traceback (most recent call last):
  File "graph_error.py", line 6, in <module>
    uhd.rfnoc.RfnocGraph("addr=localhost")
RuntimeError: RuntimeError: Failure to create rfnoc_graph.
12
[INFO] [UHD] linux; GNU C++ version 9.2.0; Boost_107100; UHD_4.2.0.0-0-g46a70d85
[INFO] [MPMD] Initializing 1 device(s) in parallel with args: 
mgmt_addr=127.0.0.1,type=x4xx,product=x410,serial=323F754,name=NE-LAB-X
410-01,fpga=X4_400,claimed=False,addr=localhost
[INFO] [MPM.PeriphManager] init() called with device args 
`fpga=X4_400,mgmt_addr=127.0.0.1,name=NE-LAB-X410-01,product=x410,clock_sou
rce=internal,time_source=internal'.
[ERROR] [RFNOC::GRAPH] IO Error during GSM initialization. EnvironmentError: 
IOError: Timed out getting recv buff for management tran
saction
[ERROR] [RFNOC::GRAPH] Caught exception while initializing graph: 
EnvironmentError: IOError: Timed out getting recv buff for manageme
nt transaction
Traceback (most recent call last):
  File "graph_error.py", line 6, in <module>
    uhd.rfnoc.RfnocGraph("addr=localhost")
RuntimeError: RuntimeError: Failure to create rfnoc_graph.
13
[INFO] [UHD] linux; GNU C++ version 9.2.0; Boost_107100; UHD_4.2.0.0-0-g46a70d85
[INFO] [MPMD] Initializing 1 device(s) in parallel with args: 
mgmt_addr=127.0.0.1,type=x4xx,product=x410,serial=323F754,name=NE-LAB-X
410-01,fpga=X4_400,claimed=False,addr=localhost
[INFO] [MPM.PeriphManager] init() called with device args 
`fpga=X4_400,mgmt_addr=127.0.0.1,name=NE-LAB-X410-01,product=x410,clock_sou
rce=internal,time_source=internal'.
[ERROR] [RFNOC::GRAPH] IO Error during GSM initialization. EnvironmentError: 
IOError: Timed out getting recv buff for management tran
saction
[ERROR] [RFNOC::GRAPH] Caught exception while initializing graph: 
EnvironmentError: IOError: Timed out getting recv buff for manageme
nt transaction



Maximilian Matthe

Head of Engineering Lab

maximilian.mat...@barkhauseninstitut.org

Tel.: +49 173 4509667


Barkhausen Institut
www.barkhauseninstitut.org


Barkhausen Institut gGmbH | Sitz: Würzburger Straße 46, 01187 Dresden, Germany 
| Registergericht: Amtsgericht Dresden, HRB 37267 | Geschäftsführer: Prof. Dr. 
Gerhard Fettweis, Dr. Tim Hentschel | Vorsitzender der 
Gesellschafterdelegation: Dr. Andreas Handschuh

Hinweise zum Datenschutz und zur Verarbeitung Ihrer Daten finden Sie unter: 
https://barkhauseninstitut.org/data-privacy

This email and any attachments are intended only for the person to whom this 
email is addressed and may contain confidential and/or privileged information. 
If you received this email in error, please do not disclose the contents to 
anyone, but notify the sender by return email and delete this email (and any 
attachments) from your system. Information on data protection and processing of 
your personal information: https://barkhauseninstitut.org/data-privacy

_______________________________________________
USRP-users mailing list -- usrp-users@lists.ettus.com
To unsubscribe send an email to usrp-users-le...@lists.ettus.com

[USRP-users] X410 - FPGA unresponsive after several RFNoC Graph Creations

Reply via email to