Hi all,
Thanks a lot Sam and Marcus for replying so quickly.
Le me first include Marcus answer in this email chain for a cleaner
communication:
"*Gnu Radio cannot really misalign samples, unless the DSP logic arranges
for that to happen. If you have a block with a bunch of streams, that
blocks "work" function cannot proceed until there is equal amounts of data
available on all streams. In the absence of the hardware dropping samples,
coherence is maintained in that case. What you are seeing here seems to be
a problem with your network stack losing things. What type of Ethernet
adapters do you have? What sample rates are you running?*"
*The requested extra info:*
- The sampling rates selected are *1Msps* for the TX and *2Msps* for the RX.
- The SFP connector attached to the USRP-X310s: AVAGO, 1.25GBd, 1000BASE-T,
ABCU-5730ARZ.
- The NIC: Intel Corporation Gigabit CT Desktop Adapter.
- The Ethernet controller: Intel Corporation 82574L Gigabit Network
Connection.
@Marcus,
good thing that the scheduler in each block handles that correctly. I'll
focus on the network setup then.
@Sam,
it seems reasonable to think that the source of error is the poking
timeout, being the misalignment the aftermath.
When you say flow controls, do you mean the ones handled by the UHD fw
functions? I see a couple of D's (packet drops) right before the
misalignment error, after the UHD raises the last poking exception. But
maybe this is just the natural behavior- to drop misaligned packets before
either getting an alignment or timing out and rising the exception.
"*Drop the sample rate (what is it, by the way?) and see if there's a
threshold where things start working [3]*":
Should I drop the samples below 1Msps/2Msps? I think these are pretty low
and having lower sampling rates would rise new problems. I can go ahead and
try it if you still thing this is the issue.
"*Keep the original sample rate and try removing a radio or two from the
system. Does this help things?*"*:*
- 4RX/1TX with simplistic flowgraph (usrp_source connected to file_sinks
and analog signal connected to usrp_sink): OK.
- 6RX/1TX with simplistic flowgraph (usrp_source connected to file_sinks
and analog signal connected to usrp_sink): NOK (error described previously).
- 3RX/1TX with my DSP blocks: OK.
- 4RX/1TX with my DSP blocks: NOK (error described previously).
Regarding an example, I could not find one that sets up a usrp_source AND a
usrp_sink, in the folder {gr_prefix}/gr-uhd/examples/. I created a
simplistic python flowgraph that results in the poking and misalignment
issue, attached in this email. For instance, the 6x1 connections would map
to the input argumments --tx_channels 1 --rx_channels 6.
Thanks a lot for your time and I hope to hear back from you soon.
Best,
Carlos
On Fri, Oct 18, 2019 at 1:05 PM Sam Reiter <sam.rei...@ettus.com> wrote:
> Carlos,
>
> On the host side if you're using a single streamer, it will work to
> time-align samples from every USRP as they come in. The reason you see 1002
> received packets without a timestamp match is because 1000 was the
> threshold for maximum number of alignment failures [1]. This can be
> changed or increased with [2]:
>
> set_alignment_failure_threshold(1000);
>
> But I doubt this will solve the issue for you. Likely just delay it. From the
> errors you included, I'd say the real problem is:
>
>
>> *ERROR] [X300] 192.168.100.2 <http://192.168.100.2/>: x300 fw
>> communication failure #1EnvironmentError: IOError: x300 fw poke32 - reply
>> timed out*
>>
>
> There is either a backup or a lack of samples coming from the radio. Do
> you see flow control errors leading up to this failure?
>
> I'd say that pulling pieces out of this system would be a good way to
> start troubleshooting:
>
> - Drop the sample rate (what is it, by the way?) and see if there's a
> threshold where things start working [3]
> - Keep the original sample rate and try removing a radio or two from
> the system. Does this help things?
>
> It sounds like this is something you can reproduce independent of your DSP
> blocks, which helps simplify things a bit. If you could go a bit further
> and find a simple example (i.e. take one of the GNURadio UHD shipping
> examples and expand it to use 3x USRPs) and get it to reproduce the
> behavior, then I'd be happy to set something up on my end try to reproduce
> the behavior you're seeing.
>
> Sam
>
> [1]
> http://lists.ettus.com/pipermail/usrp-users_lists.ettus.com/2017-August/053986.html
> [2]
> http://lists.ettus.com/pipermail/usrp-users_lists.ettus.com/2014-January/036210.html
> [3]
> http://lists.ettus.com/pipermail/usrp-users_lists.ettus.com/2014-November/039561.html
>
>
> On Thu, Oct 17, 2019 at 3:46 PM Carlos Bocanegra via USRP-users <
> usrp-users@lists.ettus.com> wrote:
>
>> Hello community,
>>
>> I am working on a gnuradio application that synchronously retrieves data
>> from a usrp_source, does some DSP on each of the streams and selects the
>> signal to send on a usrp_sink based on some metric computation. The sources
>> and sinks represent 3 USRP X310 devices, each with 2 UBX-160
>> daughterboards, and are synchronized using an Octoclock-G. Thus, creating a
>> 6x1 MISO system. The USRP synchronization is done using the Python API, as
>> well as the flowgraph. The DSP blocks are coded in C++. The flowgraph would
>> be something like:
>>
>> *self.connect((usrp_source, 0), my_dsp_0, (my_switch, 0))*
>> *self.connect((usrp_source, 1), my_dsp_1, (my_switch, 1))*
>> *self.connect((usrp_source, 2), my_dsp_2, (my_switch, 2))*
>> *self.connect((usrp_source, 3), my_dsp_3, (my_switch, 3))*
>> *self.connect(my_switch, another_dsp_0, (usrp_sink, 0))*
>>
>> The application runs for a while until UHD outputs a message saying that
>> the poke32 has timed out on one of the USRP. After that, a misalignment
>> error on the receiver streams is shown and the green lights on the USRP go
>> off.
>>
>> We are under the impression that the DSP blocks working independently may
>> be requesting samples from the source at different times, causing the
>> misalignment. Could this be the case? Is there an issue with my network
>> connection?
>>
>> I have tried to get rid of the misalignment error by increasing the
>> threshold in the file
>> *{uhd_prefix}/host/lib/transport/super_recv_packet_handler.hpp, *through
>> the function *set_alignment_failure_threshold.* That got rid of the
>> misalignment error, but the failure coming from the poke function is still
>> there.
>>
>> Interestingly, the exact same behavior arises when I just connect
>> file_sinks to the outputs from the usrp_source and a simple analog signal
>> to the usrp_sink. Below the flowgraph:
>>
>> *self.connect((usrp_source, 0), file_sink_0)*
>> self.connect((usrp_source, 1), file_sink_1)
>> self.connect((usrp_source, 2), file_sink_2)
>> self.connect((usrp_source, 3), file_sink_3)
>> *self.connect(analog_sig_c, (usrp_sink, 0))*
>>
>> I'd appreciate any help on this since I'm running out of ideas here.
>> Please, find below more details.
>>
>> *The error:*
>>
>> * [ERROR] [X300] 192.168.100.2 <http://192.168.100.2>: x300 fw
>> communication failure #1EnvironmentError: IOError: x300 fw poke32 - reply
>> timed out*
>>
>> * [ERROR] [X300] 192.168.100.2 <http://192.168.100.2>: x300 fw
>> communication failure #1EnvironmentError: IOError: x300 fw poke32 - reply
>> timed out*
>>
>> * [ERROR] [X300] 192.168.100.2 <http://192.168.100.2>: x300 fw
>> communication failure #3EnvironmentError: IOError: x300 fw poke32 - reply
>> timed out*
>> [ERROR] [UHD] An unexpected exception was caught in a task loop.The task
>> loop will now exit, things may not work.EnvironmentError: IOError:
>> 192.168.100.2: x300 fw communication failure #3
>>
>>
>>
>> * [ERROR] [STREAMER] The receive packet handler failed to time-align
>> packets.1002 received packets were processed by the handler.However, a
>> timestamp match could not be determined.*
>>
>> *Here are some details of my network configuration:*
>> - UHD built from source, tag v3.14.1.1.
>> - Gnuradio built from source, tag 3.7.13.5.
>> - X310 USRP, each equipped with two UBX-160 daughterboards.
>> - The communication is over Ethernet using the 1Gb port on the X310.
>> - Each USRP is connected to a dedicated NIC.
>> - The MTU is set to 1500 for the all interfaces.
>> - The USRPs are synchronized using an Octoclock-G and the parameters
>> tuned in the Python script. The procedure is similar to the one followed in
>> benchmark_rate (uhd), for the usrp_source and usrp_sink.
>> - The NIC ring buffers are set to the maximum for both TX and RX, to
>> 4096, using ethtool.
>> - The OS ring buffers are set as suggested bu Ettus, using the following
>> commands:
>>
>>
>>
>> *>> sudo sysctl -w net.core.rmem_max=33554432 >> sudo sysctl -w
>> net.core.wmem_max=33554432 >> sudo sysctl -w
>> net.core.rmem_default=33554432 >> sudo sysctl -w
>> net.core.wmem_default=33554432*
>>
>> Best,
>> Carlos
>> _______________________________________________
>> USRP-users mailing list
>> USRP-users@lists.ettus.com
>> http://lists.ettus.com/mailman/listinfo/usrp-users_lists.ettus.com
>>
>
# Developed by: Carlos Bocanegra
from gnuradio import gr, uhd, blocks, analog
import time
import sys
import os
import optparse
from gnuradio.uhd.uhd_swig import time_spec_t
import signal
class reader_top_block(gr.top_block):
def init_multi_usrp(self, options):
self.sink = uhd.usrp_sink(
device_addr=options.usrp_address_sink,
stream_args=uhd.stream_args(
cpu_format="fc32",
channels=options.tx_channels,
dboard_clock_rate=20e6,
skip_dram=1, # faster
ignore_cal_file="",
),
)
self.source = uhd.usrp_source(
device_addr=options.usrp_address_source,
stream_args=uhd.stream_args(
cpu_format="fc32",
channels=options.rx_channels,
dboard_clock_rate=20e6,
skip_dram=1, # faster
ignore_cal_file="",
),
)
def setup_multi_usrp(self, options):
# *** USEFUL PARAMS
sink_n_Mboards = self.sink.get_num_mboards()
source_n_Mboards = self.source.get_num_mboards()
n_db_per_Mboard = 2;
# *** EXTERNAL REF
print "[UHD] Setting External Ref + PPS sources..."
for mb_idx in range(source_n_Mboards):
self.source.set_clock_source("external", mb_idx) # Mboard=0 default
self.source.set_time_source("external", mb_idx) # Mboard=0 default
if len(options.rx_channels)/((mb_idx+1)*n_db_per_Mboard)>=1:
self.source.set_subdev_spec("A:0 B:0", mb_idx) # Mboard=0 default
else:
self.source.set_subdev_spec("A:0", mb_idx) # Mboard=0 default
for mb_idx in range(sink_n_Mboards):
self.sink.set_clock_source("external", mb_idx) # Mboard=0 default
self.sink.set_time_source("external", mb_idx) # Mboard=0 default
if len(options.tx_channels)/((mb_idx+1)*n_db_per_Mboard)>=1:
self.sink.set_subdev_spec("A:0 B:0", mb_idx) # Mboard=0 default
else:
self.sink.set_subdev_spec("A:0", mb_idx) # Mboard=0 default
# *** WAIT
time.sleep(2.0)
print "[UHD] Checking if the USRPs see the Reference signal..."
for mb_idx in range(sink_n_Mboards):
print "[UHD_TX] Mboard {0} locked: to external reference? {1}".format(mb_idx,self.sink.get_mboard_sensor("ref_locked", mb_idx).to_bool())
for mb_idx in range(source_n_Mboards):
print "[UHD_RX] Mboard {0} locked: to external reference? {1}".format(mb_idx,self.source.get_mboard_sensor("ref_locked", mb_idx).to_bool())
# *** SYNCHRONIZING DEVICES
print "[PPS] PPS set to unknown source (SMA connection)"
self.sink.set_time_unknown_pps(uhd.time_spec()) # No Mboard spec
self.source.set_time_unknown_pps(uhd.time_spec()) # No Mboard spec
# *** WAIT
time.sleep(2.0)
# *** SEE PPS SYNCHRONIZATION LEVEL
curr_hw_time = self.sink.get_time_last_pps()
while curr_hw_time==self.sink.get_time_last_pps():
pass
time.sleep(0.05) # Sleep for 50ms
print "[PPS] Re-set all PPS initial values to 0"
self.sink.set_time_next_pps(uhd.time_spec_t(0.0)) # Need to put everything a zero for time alignment
self.source.set_time_next_pps(uhd.time_spec_t(0.0)) # Need to put everything a zero for time alignment
# *** WAIT
time.sleep(1.0) # Sleep for 2s
# *** SAMPLE RATE + ANTENNA CONFIG
print "[UHD] Setting Sample Rate and Antenna configuration..."
self.sink.set_samp_rate(options.dac_rate) # No Mboard spec
self.source.set_samp_rate(options.adc_rate) # No Mboard spec
for db_idx in options.tx_channels:
self.sink.set_gain(options.tx_gain, db_idx) # Dboard=0 default
self.sink.set_antenna("TX/RX", db_idx) # Dboard=0 default
print "[UHD_TX] Channel {0}: Using antenna TX/RX, Gain {1}".format(db_idx,options.rx_gain)
for db_idx in options.rx_channels:
self.source.set_gain(options.rx_gain, db_idx) # Dboard=0 default
self.source.set_antenna("RX2", db_idx) # Dboard=0 default
self.source.set_auto_dc_offset(False, db_idx) # Dboard - Uncomment this line for SBX daughterboard
print "[UHD_RX] Channel {0}: Using antenna RX2, Gain {1}, DC-offset {2}".format(db_idx,options.rx_gain,False)
# *** WAIT
time.sleep(1.0)
# *** GET TIMES
now_sink = self.source.get_time_now()
cmd_time_sink = now_sink + uhd.time_spec(0.2) # after 200ms
cmd_time_source = now_sink + uhd.time_spec(0.2) # after 200ms
# *** SET COMMAND TIME
try:
for mb_idx in range(source_n_Mboards):
self.source.set_command_time(cmd_time_sink, mb_idx)
for mb_idx in range(sink_n_Mboards):
self.sink.set_command_time(cmd_time_source, mb_idx)
except RuntimeError:
sys.stderr.write('[UHD] [WARNING] Failed to set command times on USRPs.\n')
# *** SET CENTER FREQUENCIES
print "[UHD_FREQ] Creating tune request"
treq = uhd.tune_request(options.freq)
for db_idx in options.tx_channels:
res_sink = self.sink.set_center_freq(treq, db_idx) # Dboard=0 default
for db_idx in options.rx_channels:
res_source = self.source.set_center_freq(treq, db_idx) # Dboard=0 default
while not self.sink.get_sensor("lo_locked").to_bool():
print "waiting"
pass
while not self.source.get_sensor("lo_locked").to_bool():
print "waiting"
pass
# *** WAIT
time.sleep(2.0) # after 1s, give time to the hardware to set the tuning freq
# *** CLEAR COMMAND TIMES
for mb_idx in range(sink_n_Mboards):
self.sink.clear_command_time(mb_idx)
for mb_idx in range(source_n_Mboards):
self.source.clear_command_time(mb_idx)
def __init__(self):
gr.top_block.__init__(self)
rt = gr.enable_realtime_scheduling()
##################################################
# Variables
##################################################
options.dac_rate = 1e6 # DAC rate
options.adc_rate = 2e6 # ADC rate
options.tx_channels = range(TXCHANNELS)
options.rx_channels = range(RXCHANNELS)
options.ampl = 0.5
options.tx_gain = 20
options.rx_gain = 20
options.freq = 910e6
##################################################
# Blocks
##################################################
self.file_sink_source = ["" for x in options.rx_channels]
self.null_sink_c = ["" for x in options.rx_channels]
self.null_sink_f = ["" for x in options.rx_channels]
fileNameRoot = "/tmp/source"
for i in options.rx_channels:
fileName = fileNameRoot + str(i)
self.file_sink_source[i] = blocks.file_sink(gr.sizeof_gr_complex * 1, fileName, False)
self.null_sink_c[i] = blocks.null_sink(gr.sizeof_gr_complex * 1)
self.null_sink_f[i] = blocks.null_sink(gr.sizeof_float * 1)
self.an_sig_source = analog.sig_source_f(options.dac_rate, analog.GR_SIN_WAVE, 350e3, 1, 0)
self.to_complex = blocks.float_to_complex()
##################################################
# USRP SETUP (SINKS AND SOURCES)
##################################################
if len(options.rx_channels)<=2:
options.usrp_address_source = "addr0=192.168.60.2,recv_frame_size=1472, send_frame_size=1472" # one USRP
options.usrp_address_sink = "addr=192.168.60.2,recv_frame_size=1472, send_frame_size=1472" # one USRP
elif len(options.rx_channels)<=4:
options.usrp_address_source = "addr0=192.168.60.2, addr1=192.168.50.2,recv_frame_size=1472"
options.usrp_address_sink = "addr0=192.168.60.2, addr1=192.168.50.2,send_frame_size=1472"
elif len(options.rx_channels)<=6:
options.usrp_address_source = "addr0=192.168.60.2,addr1=192.168.50.2,addr2=192.168.100.2,recv_frame_size=1472"
options.usrp_address_sink = "addr0=192.168.60.2,addr1=192.168.50.2,addr2=192.168.100.2,send_frame_size=1472"
######## SOURCES and SINKS #########
self.init_multi_usrp(options)
self.setup_multi_usrp(options)
##################################################
# Connections
##################################################
for i in options.rx_channels:
# self.connect((self.source, i), self.null_sink_c[i]) # (Do not comment this line)
self.connect((self.source, i), self.file_sink_source[i]) # passive mode - store samples
self.connect(self.an_sig_source, self.to_complex)
for j in options.tx_channels:
self.connect(self.to_complex, (self.sink, j))
if __name__ == '__main__':
def signal_handler(sig, frame):
print('Signal captured, ending execution')
parser = optparse.OptionParser()
parser.add_option('-x', '--tx_channels', type="int", dest="TXCHANNELS", help="Specify the number of Tx channels", default=1)
parser.add_option('-y', '--rx_channels', type="int", dest="RXCHANNELS", help="Specify the number of Rx channels", default=1)
options, args = parser.parse_args()
TXCHANNELS = options.TXCHANNELS
RXCHANNELS = options.RXCHANNELS
print('Initializing Flowgraph. Press Ctrl+C to stop the Execution')
print('')
main_block = reader_top_block()
print "Starting flow..."
print('')
main_block.start()
signal.signal(signal.SIGINT, signal_handler)
signal.pause()
main_block.stop()
_______________________________________________
USRP-users mailing list
USRP-users@lists.ettus.com
http://lists.ettus.com/mailman/listinfo/usrp-users_lists.ettus.com
