Hi Nick, Good point with Wireshark and coincidentially I tried exactly that today. Both captures were fairly identical. But what's worse, things started to become more unreproducible (for example, depending on which blocks I add).
Again, what not works is that timed commands sometimes are not executed when they are supposed to be. I execute them periodically (e.g. every 100ms, 10ms, 1ms), then capture the response and slice them into equal blocks in MATLAB. The action of the timed commands (for example, retune or gain change) should always happen at the same time. Instead a few of them (maybe 3%) happen sometime in between and mess up verything. How would one go to debug this without having to acquire deep Verilog knowledge? I am 99.99% positive that the timed commands I issue via tx_command tag are correct. Are there definite answers to the questions: 1.) How many commands do the command queues have exactly on the X310? (https://kb.ettus.com/Synchronizing_USRP_Events_Using_Timed_Commands_in_UHD says 5-8 but other sources say 16 or 32). 2.) What happens when the command queue is full? Is the command dropped, does USRP crash or anything else? (above link is unclear about that) Can we change this behavior? 3.) How can we figure out what is the maximum speed to issue timed commands reliably? What does this depend on and which parameters need to be tweaked? For example, what is the fastest rate I can issue timed commands (ignoring settling times etc) on a X310 over 10Gbe? I am thinking that maybe I am sending timed commands too fast such that the command queue overflows. However: a.) How can an overflowing command queue explain that timed commands not being executed on time? b.) I played around setting send_buff_size to different values, down to send_buff_size=10240 (10kB). If I understand correctly, that means that the USRP should never receive more than 10kB data at once (one chunk holds 10240/(10e6*2*2)=256us at 10MS/s which means the if each frame has one timed command that should support sending a timed command every 256us). Unfortunately nothing changes here. Using tx_command tag in USRP Source should ensure that the commands never arrive late on the FPGA (otherwise the data would also arrive late and I would get LLLLLLL's but I don't). I thought the most straight forward would be to debug the FPGA itself but that seems to open more issues that it solves. Thanks, Lukas Gesendet: Mittwoch, 10. Juni 2020 um 19:18 Uhr Von: "Nick Foster" <bistrom...@gmail.com> An: "Lukas Haase" <lukasha...@gmx.at> Betreff: Re: [USRP-users] How to debug timed commands on FPGA side? I'd start by using Wireshark. There's a dissector for CHDR packets included in UHD (uhd/tools/dissectors). Wireshark has excellent filtering and analysis tools. You should be able to see your C++-timestamped packet go out, and compare it to the Python generated version. This will take a couple of hours to set up, run, and analyze. Following that, if you still can't find a difference, recompile an FPGA image which includes an ILA core, hooked up to some radio signals. Try to avoid having to do this, because it will take a long time, and you'll get very deep into the radio core. On the plus side, after a week or so of digging, you'll have a much better understanding of how RFNoC works under the hood. =) Nick On Wed, Jun 10, 2020 at 1:47 PM Lukas Haase via USRP-users <usrp-users@lists.ettus.com[mailto:usrp-users@lists.ettus.com]> wrote:Just some additional info: I enabled the maximum possible debug on the host (UHD_LOG_CONSOLE_LEVEL=trace and debug_level = debug in .gnuradio/config.conf) and sent both versions to a file. Again, the diff is identical! (This debug contains the debug messages from gr-uhd but uhd itself does not seem to generate any debug/trace messages for timed commands). Is there a way to somehow report back to the host when the command queue overflows or a timed command could not be processed at the planned time (late command)? According to https://kb.ettus.com/Synchronizing_USRP_Events_Using_Timed_Commands_in_UHD[https://kb.ettus.com/Synchronizing_USRP_Events_Using_Timed_Commands_in_UHD] "An overflow of the command queue will result in a system halt and often requires a physical reset of the FPGA.". This does not sound something that should just be silently dropped! This works for data streams so shouldn't it work for timed commands too? The USRP feels like a black box ... commands are being sent but I have no idea what happens inside or if they are even executed (except, of course, things are "not working") > Gesendet: Mittwoch, 10. Juni 2020 um 02:30 Uhr > Von: "Lukas Haase" <lukasha...@gmx.at[mailto:lukasha...@gmx.at]> > An: "USRP-userslists.ettus.com[http://USRP-userslists.ettus.com]"; > <usrp-users@lists.ettus.com[mailto:usrp-users@lists.ettus.com]> > Betreff: How to debug timed commands on FPGA side? > > Hello, > > Is there any (somewhat straight forward) way to debug timed commands on the > FPGA? > In particular, I want to know: > 1.) if any timed command is not executed as timed command but right away > (because it had a timestamp but the command was late so it was executed right > away) > 2.) if any command queue overruns > 3.) Any other sort of information that causes timed commands to misbehave > > I use "tx_command" for USRP Sink to send timed commands. The "tx_command" > tags are injected with an embedded python block. I use "Tag Debug" and save > all tags to a text file. Works. > Then, in exactly the same block diagram, I replace the embedded python block > with my C++ implementation that generates tags. > Suddenly, some timed commands do not execute at the right moment any more > (these are just few and consistent across re-runs and reboots). > > However, the tags generated by boths blocks are absolutely IDENTICAL! A diff > between the "tx_command" outputs results in NO differences. Hence I need to > know what the FPGA actually processes in both cases. > > Thanks > Lukas > > _______________________________________________ USRP-users mailing list USRP-users@lists.ettus.com http://lists.ettus.com/mailman/listinfo/usrp-users_lists.ettus.com
