Hi Marcus, > On 06/10/2020 09:00 PM, Lukas Haase via USRP-users wrote: >> Hi Nick, >> >> Good point with Wireshark and coincidentially I tried exactly that >> today. Both captures were fairly identical. But what's worse, >> things started to become more unreproducible (for example, >> depending on which blocks I add). >> >> Again, what not works is that timed commands sometimes are not >> executed when they are supposed to be. I execute them periodically >> (e.g. every 100ms, 10ms, 1ms), then capture the response and slice >> them into equal blocks in MATLAB. The action of the timed commands >> (for example, retune or gain change) should always happen at the >> same time. Instead a few of them (maybe 3%) happen sometime in >> between and mess up verything. >> >> How would one go to debug this without having to acquire deep >> Verilog knowledge? I am 99.99% positive that the timed commands I >> issue via tx_command tag are correct. >> >> Are there definite answers to the questions: >> >> 1.) How many commands do the command queues have exactly on the >> X310? >> (https://kb.ettus.com/Synchronizing_USRP_Events_Using_Timed_Commands_in_UHD >> says 5-8 but other sources say 16 or 32). 2.) What happens when the >> command queue is full? Is the command dropped, does USRP crash or >> anything else? (above link is unclear about that) Can we change >> this behavior? > Something to consider is that a tuning command is NOT, in general, a > single register-set to the FPGA. The FPGA has NO 'inherent' > knowledge of the various types of peripherals involved in tuning. So > tuning will often involve more than one SPI or I2C interaction with > whatever RF hardware is being used. This invariably means more than > one "command" from the FPGAs point-of-view.
Yes, I am aware of this. I deduce from Jonathon's response that the mapping of "high level command" to "register writes" (which may eventually result in the SPI/I2C interactions) happens in uhd on the host. I just can't find where this is happening. Example: tuning requests eventually call tune_xx_subdev_and_dsp from multi_usrp and set values in dsp_subtree and rf_fe_subtree. But where are these fetched, translated to register addresses/values (chdr.cmd.addr, chdr.cmd.val) etc? >> 3.) How can we figure out what is the maximum speed to issue timed >> commands reliably? What does this depend on and which parameters >> need to be tweaked? >> >> For example, what is the fastest rate I can issue timed commands >> (ignoring settling times etc) on a X310 over 10Gbe? > That partially depends on things like which SPI/I2C peripherals are > involved--those buses run at a fixed and not-steaming-fast rate. So > even though the FPGA may be able to swallow commands "real fast", it > won't necessarily be able to complete SPI/I2C bus transactions that > fast. > > My recollection is that the SPI on the X310 operates at perhaps > 1Mbit/sec, but maybe even as sluggish as 400kbit/s. > > So, when UHD on the host side is tuning a hardware device it doesn't > say to the FPGA "hey, could you tune the UBX card to <X> MHz, > please?" It says "hey, here's an SPI transaction. Oh, and another > one. Oh, and maybe another one, too..." As I have written in my original post "ignoring analog settlings etc". For now it's really about making sure that the *FPGA* actually executes stuff when it should and that seems to fail (even for a pure DSP retune). Best, Lukas >> I am thinking that maybe I am sending timed commands too fast such >> that the command queue overflows. However: a.) How can an >> overflowing command queue explain that timed commands not being >> executed on time? b.) I played around setting send_buff_size to >> different values, down to send_buff_size=10240 (10kB). If I >> understand correctly, that means that the USRP should never receive >> more than 10kB data at once (one chunk holds 10240/(10e6*2*2)=256us >> at 10MS/s which means the if each frame has one timed command that >> should support sending a timed command every 256us). Unfortunately >> nothing changes here. >> >> Using tx_command tag in USRP Source should ensure that the commands >> never arrive late on the FPGA (otherwise the data would also arrive >> late and I would get LLLLLLL's but I don't). >> >> I thought the most straight forward would be to debug the FPGA >> itself but that seems to open more issues that it solves. >> >> Thanks, Lukas >> >> >> >> Gesendet: Mittwoch, 10. Juni 2020 um 19:18 Uhr Von: "Nick Foster" >> <bistromath at gmail.com> An: "Lukas Haase" <lukashaase at gmx.at> >> Betreff: Re: [USRP-users] How to debug timed commands on FPGA >> side? >> >> I'd start by using Wireshark. There's a dissector for CHDR packets >> included in UHD (uhd/tools/dissectors). Wireshark has excellent >> filtering and analysis tools. You should be able to see your >> C++-timestamped packet go out, and compare it to the Python >> generated version. This will take a couple of hours to set up, run, >> and analyze. >> >> Following that, if you still can't find a difference, recompile an >> FPGA image which includes an ILA core, hooked up to some radio >> signals. Try to avoid having to do this, because it will take a >> long time, and you'll get very deep into the radio core. On the >> plus side, after a week or so of digging, you'll have a much better >> understanding of how RFNoC works under the hood. =) >> >> Nick >> >> On Wed, Jun 10, 2020 at 1:47 PM Lukas Haase via USRP-users >> <usrp-users at lists.ettus.com[mailto:usrp-users at >> lists.ettus.com]> wrote:Just some additional info: I enabled the >> maximum possible debug on the host (UHD_LOG_CONSOLE_LEVEL=trace and >> debug_level = debug in .gnuradio/config.conf) and sent both >> versions to a file. Again, the diff is identical! (This debug >> contains the debug messages from gr-uhd but uhd itself does not >> seem to generate any debug/trace messages for timed commands). >> >> Is there a way to somehow report back to the host when the command >> queue overflows or a timed command could not be processed at the >> planned time (late command)? >> >> According to >> https://kb.ettus.com/Synchronizing_USRP_Events_Using_Timed_Commands_in_UHD[https://kb.ettus.com/Synchronizing_USRP_Events_Using_Timed_Commands_in_UHD] >> "An overflow of the command queue will result in a system halt and >> often requires a physical reset of the FPGA.". This does not sound >> something that should just be silently dropped! >> >> This works for data streams so shouldn't it work for timed commands >> too? >> >> The USRP feels like a black box ... commands are being sent but I >> have no idea what happens inside or if they are even executed >> (except, of course, things are "not working") >> >> >>> Gesendet: Mittwoch, 10. Juni 2020 um 02:30 Uhr Von: "Lukas Haase" >>> <lukashaase at gmx.at[mailto:lukashaase at gmx.at]> An: >>> "USRP-userslists.ettus.com[http://USRP-userslists.ettus.com]"; >>> <usrp-users at lists.ettus.com[mailto:usrp-users at >>> lists.ettus.com]> Betreff: How to debug timed commands on FPGA >>> side? >>> >>> Hello, >>> >>> Is there any (somewhat straight forward) way to debug timed >>> commands on the FPGA? In particular, I want to know: 1.) if any >>> timed command is not executed as timed command but right away >>> (because it had a timestamp but the command was late so it was >>> executed right away) 2.) if any command queue overruns 3.) Any >>> other sort of information that causes timed commands to >>> misbehave >>> >>> I use "tx_command" for USRP Sink to send timed commands. The >>> "tx_command" tags are injected with an embedded python block. I >>> use "Tag Debug" and save all tags to a text file. Works. Then, in >>> exactly the same block diagram, I replace the embedded python >>> block with my C++ implementation that generates tags. Suddenly, >>> some timed commands do not execute at the right moment any more >>> (these are just few and consistent across re-runs and reboots). >>> >>> However, the tags generated by boths blocks are absolutely >>> IDENTICAL! A diff between the "tx_command" outputs results in NO >>> differences. Hence I need to know what the FPGA actually >>> processes in both cases. >>> >>> Thanks Lukas >>> >>> >> >> >> _______________________________________________ USRP-users mailing >> list USRP-users at lists.ettus.com >> http://lists.ettus.com/mailman/listinfo/usrp-users_lists.ettus.com _______________________________________________ USRP-users mailing list USRP-users@lists.ettus.com http://lists.ettus.com/mailman/listinfo/usrp-users_lists.ettus.com
