Hi Ian,
To answer your questions:
- The absolute times after run are around the 6 second mark as expected,
just with some additional 10s to 100s of microseconds that appears random
on each B210.
- All of the B210s are powered by the 5.9V 4.0A adapter they came with.
Now some good news: I think I've found a workaround. In my minimal C++
example, I first narrowed it down to get_rx_stream() where the device time
was changing. I found that in b200_impl::update_enables() that the calls
to set_active_chains(), specifically when the RX chains are turned on/off,
as well as the call to update_gpio_state() both cause the device time to
lose time sync. After trying to find a fix there I just did something more
simple: moved the synchronization code to after the call
to get_rx_stream(). This appears to fix the problem, but requires some
non-obvious steps if using GNU Radio (specially, need to call start/stop on
the USRP blocks to create the RX stream, and then do the synchronization).
Thanks for the insight on the possible time misalignment between multiple
B210s. Just to make sure I understand - for two boards with the same master
clock setting and same external PPS/10 MHz reference, there could be up to
1 master clock cycle of excess delay between the two? But on the N2xx or
X3xx that would not be the case? If so, that is useful to know, since I'm
attempting to use the B210 for time of arrival measurements and 1 clock
cycle could be significant.
-Michael
On Thu, Feb 27, 2020 at 2:36 AM Ian Buckley <i...@ionconcepts.com> wrote:
> This is a particularly curious problem.
> The offset between B210’s of the “time after run” values after sample
> capture has completed is very hard to explain…certainly from a hardware
> only perspective.
> What are the absolute time values reported? Are they at least
> approximately 5-6 seconds as you might expect from your python? Or some
> completely random value?
> None of the UHD commands you use after “set_time_next_pps” should cause
> any change in the relationship of the time of day counters and the PPS.
> It makes sense that all samples have the expected RX metadata data, the
> capture is triggered by an arithmetic comparison between the time-of-day
> counters and
> the value you load into the H/W as a trigger using “set_start_time”, and
> the counter value is read to add meta data to each packet of RX samples
> that is assembled.
> Superficially it would appear that somehow the time-of-day counters are
> “corrupted” between the "time after align” values you read and before they
> count to timespec(2.0)
>
> (Completely random thought, but do you have the B210’s powered via USB or
> from the 6V jacks? I’m presuming they are plugged into to adjacent USB
> ports on a host?)
>
> I should probably caution you, unrelated to this issue that the B210 isn't
> designed so that multiple boards can be synced and used coherently unlike
> N210/X310. Whilst the master clock
> Is disciplined by the 10MHz, you are able to program it to arbitrary
> frequencies that are not phase aligned between boards which means PPS edge
> detect events as seen by the B210’s
> may occur at different actual times, though this would only produce a time
> misalignment of upto only 1 master clock period.
>
> BTW Marcus the PPS bug I *think you are referring to (edge detect driven
> by different synchronizers) was fixed before 3.9LTS was released and would
> have manifested differently.
>
> On Feb 25, 2020, at 10:35 AM, Marcus D. Leech via USRP-users <
> usrp-users@lists.ettus.com> wrote:
>
> On 02/25/2020 09:33 AM, Michael Wentz via USRP-users wrote:
>
> Marcus,
>
> Thanks for the suggestion. I added set_subdev_spec('A:A A:B') for the 2
> channel case, but the results are the same.
>
> I also tried this code with a pair of X310s receiving. The results are
> somewhat similar:
> - Channel 0 (A:0) across 2 devices: no delay
> - Channel 1 (B:0) across 2 devices: no delay
> - Channels 0 and 1 (A:0 B:0) across 2 devices: channel 0 is in sync
> between the two, but channel 1 is not - and channel 1 is not in sync with
> channel 0 on either device.
> - Unlike with the B210s, the X310s both report the same time after
> streaming.
>
> Let us focus on the B210 first.
>
> I've been through your code, and I cannot find anything wrong with it.
> Now, I recall that there was a bug in earlier UHD with the two
> clocks in the B210 (each conceptual radio block in the FPGA includes its
> own time-of-day clock), and they wouldn't always get reset
> together, even with set_time_next_pps(), so there would be some skew.
>
> This, or something related may be the root cause of your issue. The
> engineer who most recently worked on B210 is being consulted
> on this, and hopefully myself or Sam Reiter will have an answer soon.
>
>
>
> I've done this in the past with many X310s (each with 2 channels) with
> success, but that was with an older/pre-RFNoC UHD circa 2016. I'm using
> pretty much the same code here. Anything else I can try to time sync
> multiple devices?
>
> -Michael
>
> On Mon, Feb 24, 2020 at 9:07 PM Marcus D. Leech via USRP-users <
> usrp-users@lists.ettus.com> wrote:
>
>> On 02/24/2020 05:10 PM, Michael Wentz via USRP-users wrote:
>>
>> Hi,
>>
>> I'm trying to synchronize multiple B210s to receive data at the same
>> time. I'm only concerned about sample time alignment across devices, and am
>> aware there is a phase offset that will need to be calibrated out
>> separately. I've had success with 1 RX channel across 2 B210s, but
>> something seems wrong when using 2 RX channels across 2 B210s.
>>
>> I've pasted a program below to help reproduce this issue. It uses 1 B210
>> to TX noise and 2 separate B210s to RX at the same time. There is a 40 dB
>> attenuator on the output of the transmitting B210, then a 4 way splitter,
>> and equal length cables to both receivers on the other 2 B210s.
>> Additionally, the 2 receiving B210s get 10 MHz and PPS from a common
>> OctoClock-G.
>>
>> Here are 10 observations of sample offsets between devices (1 MSPS rate):
>> Only channel 0 on both RX: [0,0,0,0,0,0,0,0,0,0]
>> Only channel 1 on both RX: [0,0,0,0,0,0,0,0,0,0]
>> Channels 0 and 1 simultaneously on both
>> RX: [14,1,-126,48,2,-88,17,-204,-96,2]
>>
>> In the 2 channel case both channels on one device are always aligned, but
>> are offset from the channels on the other device. Both of the receiving
>> B210s show the same time last PPS when asked before the streams start. The
>> times in the RX metadata also match. But after the streams stop, the
>> devices respond with different times - neither of which are correct - and
>> the difference between them matches what I'm measuring.
>>
>> I'm using UHD 3.14.1.1 and GNU Radio 3.8 on RHEL 7.7. Any ideas?
>>
>> Thanks,
>> Michael
>>
>> ----------
>>
>> #!/usr/bin/env python3
>>
>> from gnuradio import gr, uhd, analog, blocks
>> import time
>> import numpy as np
>>
>> tx_serial = '30D1***'
>> rx_serials = ['3153***', '3153***']
>> rx_channels = [0, 1]
>>
>> n_rx = len(rx_serials)
>> n_chan = len(rx_channels)
>>
>> class top_block(gr.top_block):
>> def __init__(self):
>> gr.top_block.__init__(self)
>>
>> # transmit path
>> noise_src = analog.noise_source_c(analog.GR_GAUSSIAN, 0.1)
>> tx_strm_args = uhd.stream_args(cpu_format='fc32', channels=[0])
>> tx = uhd.usrp_sink('serial=' + tx_serial, tx_strm_args)
>> tx.set_clock_rate(16e6)
>> tx.set_samp_rate(1e6)
>> tx.set_center_freq(1e9)
>> tx.set_gain(40)
>> tx.set_antenna('TX/RX')
>> self.connect(noise_src, tx)
>>
>> # receive path
>> rx_strm_args = uhd.stream_args(cpu_format='fc32',
>> channels=rx_channels)
>> self.rx = [None]*n_rx
>> head = [None]*n_rx*n_chan
>> file_sink = [None]*n_rx*n_chan
>> for i in range(n_rx):
>> self.rx[i] = uhd.usrp_source('serial=' + rx_serials[i],
>> rx_strm_args)
>> self.rx[i].set_clock_rate(16e6)
>> self.rx[i].set_samp_rate(1e6)
>> self.rx[i].set_center_freq(1e9)
>> self.rx[i].set_gain(40)
>> self.rx[i].set_antenna('RX2')
>> self.rx[i].set_time_source('external')
>> self.rx[i].set_clock_source('external')
>> for j in range(n_chan):
>> ch = i*n_chan + j
>> head[ch] = blocks.head(2*gr.sizeof_float, 10000)
>> f = 'rx_%d%d.dat' % (i, j)
>> file_sink[ch] = blocks.file_sink(2*gr.sizeof_float, f)
>> file_sink[ch].set_unbuffered(True)
>> self.connect((self.rx[i], j), head[ch], file_sink[ch])
>>
>> # make sure 10 MHz ref is locked
>> for i in range(n_rx):
>> while not
>> self.rx[i].get_mboard_sensor('ref_locked').to_bool():
>> print('RX %d waiting for 10 MHz ref lock...' % i)
>> time.sleep(1)
>> print('RX %d 10 MHz ref locked' % i)
>>
>> # time sync the two receivers
>> for i in range(n_rx):
>> t = self.rx[i].get_time_last_pps().get_real_secs()
>> print('RX %d time before align: %r' % (i, t))
>> time_last_pps = self.rx[0].get_time_last_pps().get_real_secs()
>> while time_last_pps ==
>> self.rx[0].get_time_last_pps().get_real_secs():
>> time.sleep(0.1)
>> for i in range(n_rx):
>> self.rx[i].set_time_next_pps(uhd.time_spec_t(0))
>> time_last_pps = self.rx[0].get_time_last_pps().get_real_secs()
>> while time_last_pps ==
>> self.rx[0].get_time_last_pps().get_real_secs():
>> time.sleep(0.1)
>> for i in range(n_rx):
>> t = self.rx[i].get_time_last_pps().get_real_secs()
>> print('RX %d time after align: %r' % (i, t))
>>
>> # collect in the future
>> rx_time = uhd.time_spec_t(2)
>> for i in range(n_rx):
>> self.rx[i].set_start_time(rx_time)
>> self.rx[i].set_recv_timeout(3)
>>
>> # run flowgraph
>> tb = top_block()
>> tb.start()
>> time.sleep(5)
>> tb.stop()
>>
>> # print time
>> t = tb.rx[0].get_time_last_pps().get_real_secs()
>> while t == tb.rx[0].get_time_last_pps().get_real_secs():
>> time.sleep(0.1)
>> for i in range(n_rx):
>> t = tb.rx[i].get_time_last_pps().get_real_secs()
>> print('RX %d time after run: %r' % (i, t))
>>
>> # cross-correlate receivers and print delay
>> y = [None]*n_rx*n_chan
>> for i in range(n_rx):
>> for j in range(n_chan):
>> ch = i*n_chan + j
>> y[ch] = np.fromfile('rx_%d%d.dat' % (i, j), dtype=np.complex64)
>> r = np.correlate(y[0], y[ch], mode='full')
>> max_lag = np.max([len(y[0]), len(y[ch])]) - 1
>> lags = np.arange(-max_lag, max_lag+1)
>> d = lags[np.argmax(np.abs(r))]
>> print('RX %d, channel %d: %d samples' % (i, j, d))
>>
>>
>> _______________________________________________
>> USRP-users mailing
>> listUSRP-users@lists.ettus.comhttp://lists.ettus.com/mailman/listinfo/usrp-users_lists.ettus.com
>>
>> IN the two channel case, you should probably set the subdev spec to:
>>
>> A:A A:B
>>
>> I didn't set a set_rx_subdev_spec() in the code.
>>
>>
>> _______________________________________________
>> USRP-users mailing list
>> USRP-users@lists.ettus.com
>> http://lists.ettus.com/mailman/listinfo/usrp-users_lists.ettus.com
>>
>
>
> _______________________________________________
> USRP-users mailing
> listUSRP-users@lists.ettus.comhttp://lists.ettus.com/mailman/listinfo/usrp-users_lists.ettus.com
>
>
> _______________________________________________
> USRP-users mailing list
> USRP-users@lists.ettus.com
> http://lists.ettus.com/mailman/listinfo/usrp-users_lists.ettus.com
>
>
>
_______________________________________________
USRP-users mailing list
USRP-users@lists.ettus.com
http://lists.ettus.com/mailman/listinfo/usrp-users_lists.ettus.com
