On 06/10/2020 09:00 PM, Lukas Haase via USRP-users wrote:
Hi Nick,
Good point with Wireshark and coincidentially I tried exactly that today. Both
captures were fairly identical. But what's worse, things started to become more
unreproducible (for example, depending on which blocks I add).
Again, what not works is that timed commands sometimes are not executed when
they are supposed to be.
I execute them periodically (e.g. every 100ms, 10ms, 1ms), then capture the
response and slice them into equal blocks in MATLAB.
The action of the timed commands (for example, retune or gain change) should
always happen at the same time. Instead a few of them (maybe 3%) happen
sometime in between and mess up verything.
How would one go to debug this without having to acquire deep Verilog knowledge?
I am 99.99% positive that the timed commands I issue via tx_command tag are
correct.
Are there definite answers to the questions:
1.) How many commands do the command queues have exactly on the X310?
(https://kb.ettus.com/Synchronizing_USRP_Events_Using_Timed_Commands_in_UHD
says 5-8 but other sources say 16 or 32).
2.) What happens when the command queue is full? Is the command dropped, does
USRP crash or anything else? (above link is unclear about that)
Can we change this behavior?
3.) How can we figure out what is the maximum speed to issue timed commands
reliably?
What does this depend on and which parameters need to be tweaked?
For example, what is the fastest rate I can issue timed commands (ignoring
settling times etc) on a X310 over 10Gbe?
This is actually an ambiguous question. Do you mean "what is the
smallest scheduling interval for the commands that will be executed in
the future?" or "how fast can I issue commands that will ultimately
be scheduled at a later time?" In the former, that depends on the
exact nature of the commands, since they end up actually being
executed by, for example, an SPI or I2C endpoint, which operates very
very much slower than a 10GiGe interface. In the latter, my guess is
that the FPGA can swallow commands and place them on the queue
pretty-much as fast as you can issue them over 10GiG. How fast you
can do that depends very much on your host-side environment, network stack,
kernel network drivers, kernel latencies, etc.
I am thinking that maybe I am sending timed commands too fast such that the
command queue overflows.
However:
a.) How can an overflowing command queue explain that timed commands not being
executed on time?
b.) I played around setting send_buff_size to different values, down to
send_buff_size=10240 (10kB). If I understand correctly, that means that the
USRP should never receive more than 10kB data at once (one chunk holds
10240/(10e6*2*2)=256us at 10MS/s which means the if each frame has one timed
command that should support sending a timed command every 256us). Unfortunately
nothing changes here.
Using tx_command tag in USRP Source should ensure that the commands never
arrive late on the FPGA (otherwise the data would also arrive late and I would
get LLLLLLL's but I don't).
I thought the most straight forward would be to debug the FPGA itself but that
seems to open more issues that it solves.
Thanks,
Lukas
Gesendet: Mittwoch, 10. Juni 2020 um 19:18 Uhr
Von: "Nick Foster" <bistrom...@gmail.com>
An: "Lukas Haase" <lukasha...@gmx.at>
Betreff: Re: [USRP-users] How to debug timed commands on FPGA side?
I'd start by using Wireshark. There's a dissector for CHDR packets included in
UHD (uhd/tools/dissectors). Wireshark has excellent filtering and analysis
tools. You should be able to see your C++-timestamped packet go out, and
compare it to the Python generated version. This will take a couple of hours to
set up, run, and analyze.
Following that, if you still can't find a difference, recompile an FPGA image which includes an ILA core, hooked up to some radio signals. Try to avoid having to do this, because it will take a long time, and you'll get very deep into the radio core. On the plus side, after a week or so of digging, you'll have a much better understanding of how RFNoC works under the hood. =)
Nick
On Wed, Jun 10, 2020 at 1:47 PM Lukas Haase via USRP-users
<usrp-users@lists.ettus.com[mailto:usrp-users@lists.ettus.com]> wrote:Just some
additional info: I enabled the maximum possible debug on the host
(UHD_LOG_CONSOLE_LEVEL=trace and debug_level = debug in .gnuradio/config.conf) and
sent both versions to a file.
Again, the diff is identical!
(This debug contains the debug messages from gr-uhd but uhd itself does not
seem to generate any debug/trace messages for timed commands).
Is there a way to somehow report back to the host when the command queue
overflows or a timed command could not be processed at the planned time (late
command)?
According to
https://kb.ettus.com/Synchronizing_USRP_Events_Using_Timed_Commands_in_UHD[https://kb.ettus.com/Synchronizing_USRP_Events_Using_Timed_Commands_in_UHD]
"An overflow of the command queue will result in a system halt and often requires a
physical reset of the FPGA.". This does not sound something that should just be
silently dropped!
This works for data streams so shouldn't it work for timed commands too?
The USRP feels like a black box ... commands are being sent but I have no idea what
happens inside or if they are even executed (except, of course, things are "not
working")
Gesendet: Mittwoch, 10. Juni 2020 um 02:30 Uhr
Von: "Lukas Haase" <lukasha...@gmx.at[mailto:lukasha...@gmx.at]>
An: "USRP-userslists.ettus.com[http://USRP-userslists.ettus.com]";
<usrp-users@lists.ettus.com[mailto:usrp-users@lists.ettus.com]>
Betreff: How to debug timed commands on FPGA side?
Hello,
Is there any (somewhat straight forward) way to debug timed commands on the
FPGA?
In particular, I want to know:
1.) if any timed command is not executed as timed command but right away
(because it had a timestamp but the command was late so it was executed right
away)
2.) if any command queue overruns
3.) Any other sort of information that causes timed commands to misbehave
I use "tx_command" for USRP Sink to send timed commands. The "tx_command" tags are
injected with an embedded python block. I use "Tag Debug" and save all tags to a text file. Works.
Then, in exactly the same block diagram, I replace the embedded python block
with my C++ implementation that generates tags.
Suddenly, some timed commands do not execute at the right moment any more
(these are just few and consistent across re-runs and reboots).
However, the tags generated by boths blocks are absolutely IDENTICAL! A diff between the
"tx_command" outputs results in NO differences. Hence I need to know what the
FPGA actually processes in both cases.
Thanks
Lukas
_______________________________________________
USRP-users mailing list
USRP-users@lists.ettus.com
http://lists.ettus.com/mailman/listinfo/usrp-users_lists.ettus.com
_______________________________________________
USRP-users mailing list
USRP-users@lists.ettus.com
http://lists.ettus.com/mailman/listinfo/usrp-users_lists.ettus.com
