Thanks for the suggestions. We are transmitting from a file. It is on a fast SSD and retrieval rate is not an issue (it easily supports > 15 Gbps sustained average read rates). The data is dynamic and repeating is not an option.
We also produced a threaded version of tx_samples_from_file to see if that would help. It did not fix the problem, although it did seem to help. We have confirmed that the new code version is NEVER in a state of not having data to send across on the socket, although there are periods (microseconds? nanoseconds?) between consecutive USRP send calls - which cannot be avoided, as far as I can see. We have used RFNoC tools and generally have had fewer performance issues with them as compared to the stock UHD/USRP tools. However, there isn't one that supports continuous data streaming forever without repeats from a file. Your suggestion of modifying the FPGA to add buffering makes sense to me and we do have some FPGA talent, although that is not my skill. I will look into it.. Thanks! Jim At 03:37 PM 4/13/2023, Rob Kossler via USRP-users wrote: >Hi Jim, > From a quick review of this chain, it appears your concern is for the > transmit direction (hence, Underruns). Although I have never tried to run for > extended periods such as you are requiring, I am reasonably confident that > the issue is on the host side and not the Radio side. Some remarks: > * Perhaps you would get more reliable streaming using DPDK. When I last > tried this a couple of years ago, the performance was great (at much higher > rates than you, but for shorter periods of time). But, the side effects were > substantial (PC mis-behaving to the point of becoming unusable) and so I > abandoned it. The new DPDK is more mature (as is the OS) from when I tried it > so it may be much better now with regard to side effects. > * You could add buffering on the N310 (by building a custom FPGA image) > that could robustly handle short host "hiccups" in streaming. Originally, > the DMA-FIFO RFNoC block was put in the Tx path of devices such as the X310 > for this express purpose. But, the DMA-FIFO block cannot handle 4 streams at > full rate on an N310 so it is not part of the stock FPGA. The external RAM is > used instead for the "Replay" block. But at your data rates, the external > RAM can handle four streams so you could build an FPGA image that replaced > the Replay block with the DMA-FIFO such that you would have very large > buffers on the FPGA to handle host streaming hiccups. Note that you could > also build a new image with larger FPGA fabric buffers, but they wouldn't > approach the size of the external RAM. If building your own FPGA image > seems daunting (for some this is a show stopper), I just want to mention that > what I am suggesting would not require FPGA design talent - the necessary > blocks already exist - it would just require purchasing Xilinx Vivado and > getting past the initial learning curve of building an image. > * How are you generating your Tx samples? Custom app? Reading from a > very large data file? I ask this because I have had the most streaming > success when transmitting from a file (or receive to a file) if the file is > in a "RAM disk". We generally order Linux PCs with as much RAM as we can > afford in order to configure such RAM as a RAM disk for the purpose of fast > streaming to/from files. > * Finally, if your data is not dynamic (a repeated fixed length sequence > such as in a repeated radar transmission), I would highly recommend using the > Replay block to send the data rather than streaming from the host. You will > likely never see an underrun in this case. But, I realize that if the data > is dynamic, this is not possible. >Rob > > >On Thu, Apr 13, 2023 at 5:03â¯PM Jim Schatzman ><<mailto:james.schatz...@futurelabusa.com>james.schatz...@futurelabusa.com> >wrote: >Even with all the recommended settings, and a very fast computer that is doing >nothing except sending the data, it is maybe 50/50 that a 2 hour simulation >can be conducted without an underrun. The longest run I have been able to do >without an underrun is about 2.5 hours. > >The sample rate is 12.5 Msamp/sec at 16 bit I + 16 bit Q or 400 Mbit/sec. > >For our application, that is unacceptable. I need to be able to run for days >without data loss. > >It is a mystery to me why a 10 Gbit connection cannot support 400 Mbit/sec UDP >reliably. > >Any ideas about how we can completely eliminate underruns? > >At the moment, I am uncertain whether the problem is occurring on the host or >on the radio. I suspect the radio, but I will do some testing of the host to >see what UDP data rate it can support without loss. > >Thanks! > > > > > >At 10:53 PM 4/10/2023, Marcus D. Leech wrote: >>On 10/04/2023 21:12, Jim Schatzman wrote: >>>The following steps had no impact: >>> >>>a) Don't use a switch but do a point-to-point connection between the >>>comptuer's NIC and the N310. >>>b) Adjust network buffers and ring buffer per recommendations (actually, the >>>network buffers setting was always being done). >>> >>>Increasing the MTU to 9000 had a significant impact. An occasionaly underrun >>>is still experienced, but an hour or two of continuous transmission is >>>possible. >>> >>>I am wondering if this is not an issue on the computer side but on the radio >>>side. Is the N310 incapable of supporting 1x 10 Gbps streams with a MTU of >>>1500? Perhaps. >>> >>>I will do some computer-to-computer testing to see if the problem can be >>>reproduced there. >>> >>>Jim >>Even non-SDR applications tend to use jumbo-frames for continuous traffic at >>10Gbit.àI'm sorry that I didn't clue in to that >>àin the first round. >> >> >>> >>> >>> >>> >>>At 08:39 PM 4/7/2023, Marcus D. Leech wrote: >>>>On 07/04/2023 22:13, Jim Schatzman wrote: >>>>>We have been unable to estable 100% reliable connections to an N310 USRP >>>>>radio through its 10 Gbit ethernet from Linux. What happens is that it >>>>>works fine for a period of time - 30 to 60 minutes, typically. Then we see >>>>>a couple of U's in the output. Unfortunately, that is fatal for our >>>>>application. >>>>> >>>>>Using the unmodified tx_samples_from_file or one modified to use separate >>>>>threads to read data from the file and to sent it over the socket to the >>>>>radio, the symptoms are the same. >>>>> >>>>>All the evidence is that the application is sending data continuously >>>>>without any delays. Also, the "network" has no devices on it except for >>>>>the host computer, a high performance 10G switch, and the N310 radio. >>>>> >>>>>We are wondering if this could be a Linux "feature". We would like to try >>>>>increasing the socket priority with SO_PRIORITY. However, we are not >>>>>finding any hooks in the UHD software for this. >>>>> >>>>>Suggestions? >>>>> >>>>>Thanks! >>>>>Jim >>>>>_______________________________________________ >>>>Have you increased the ring buffers on your card? >>>> >>>><https://kb.ettus.com/USRP_Host_Performance_Tuning_Tips_and_Tricks#Increasing_Ring_Buffers>https://kb.ettus.com/USRP_Host_Performance_Tuning_Tips_and_Tricks#Increasing_Ring_Buffers >>>> >>>>Also, adjust the network buffers: >>>> >>>><https://kb.ettus.com/USRP_Host_Performance_Tuning_Tips_and_Tricks#Adjust_Network_Buffers>https://kb.ettus.com/USRP_Host_Performance_Tuning_Tips_and_Tricks#Adjust_Network_Buffers >>>> >>>>Have you tried a direct connection--without the switch?à à àJust >>>>to eliminate it.Ãit.àBefore you say "oh, it's not the switch", >>>>àmany "high performance" 10G swiitches have a less-than-ideal >>>>switching fabric. >>>> >>>>_______________________________________________ USRP-users mailing list -- >>>><mailto:usrp-users@lists.ettus.com>usrp-users@lists.ettus.com To >>>>unsubscribe send an email to >>>><mailto:usrp-users-le...@lists.ettus.com>usrp-users-le...@lists.ettus.com >>>></x-flowed> >>>_______________________________________________ >>>USRP-users mailing list -- >>><mailto:usrp-users@lists.ettus.com>usrp-users@lists.ettus.com >>>To unsubscribe send an email to >>><mailto:usrp-users-le...@lists.ettus.com>usrp-users-le...@lists.ettus.com >>_______________________________________________ USRP-users mailing list -- >><mailto:usrp-users@lists.ettus.com>usrp-users@lists.ettus.com To unsubscribe >>send an email to >><mailto:usrp-users-le...@lists.ettus.com>usrp-users-le...@lists.ettus.com >></x-flowed> >_______________________________________________ >USRP-users mailing list -- ><mailto:usrp-users@lists.ettus.com>usrp-users@lists.ettus.com >To unsubscribe send an email to ><mailto:usrp-users-le...@lists.ettus.com>usrp-users-le...@lists.ettus.com > >_______________________________________________ >USRP-users mailing list -- usrp-users@lists.ettus.com >To unsubscribe send an email to usrp-users-le...@lists.ettus.com
_______________________________________________ USRP-users mailing list -- usrp-users@lists.ettus.com To unsubscribe send an email to usrp-users-le...@lists.ettus.com
