On 13/04/2023 17:37, Rob Kossler wrote:
Hi Jim,
From a quick review of this chain, it appears your concern is for the
transmit direction (hence, Underruns). Although I have never tried to
run for extended periods such as you are requiring, I am reasonably
confident that the issue is on the host side and not the Radio side.
Some remarks:
* Perhaps you would get more reliable streaming using DPDK. When I
last tried this a couple of years ago, the performance was
great (at much higher rates than you, but for shorter periods of
time). But, the side effects were substantial (PC mis-behaving to
the point of becoming unusable) and so I abandoned it. The new
DPDK is more mature (as is the OS) from when I tried it so it may
be much better now with regard to side effects.
* You could add buffering on the N310 (by building a custom FPGA
image) that could robustly handle short host "hiccups" in
streaming. Originally, the DMA-FIFO RFNoC block was put in the Tx
path of devices such as the X310 for this express purpose. But,
the DMA-FIFO block cannot handle 4 streams at full rate on an N310
so it is not part of the stock FPGA. The external RAM is used
instead for the "Replay" block. But at your data rates, the
external RAM can handle four streams so you could build an FPGA
image that replaced the Replay block with the DMA-FIFO such that
you would have very large buffers on the FPGA to handle host
streaming hiccups. Note that you could also build a new image
with larger FPGA fabric buffers, but they wouldn't approach the
size of the external RAM. If building your own FPGA image seems
daunting (for some this is a show stopper), I just want to mention
that what I am suggesting would not require FPGA design talent -
the necessary blocks already exist - it would just require
purchasing Xilinx Vivado and getting past the initial learning
curve of building an image.
* How are you generating your Tx samples? Custom app? Reading from
a very large data file? I ask this because I have had the most
streaming success when transmitting from a file (or receive to a
file) if the file is in a "RAM disk". We generally order Linux
PCs with as much RAM as we can afford in order to configure such
RAM as a RAM disk for the purpose of fast streaming to/from files.
* Finally, if your data is not dynamic (a repeated fixed length
sequence such as in a repeated radar transmission), I would highly
recommend using the Replay block to send the data rather than
streaming from the host. You will likely never see an underrun in
this case. But, I realize that if the data is dynamic, this is
not possible.
Rob
I'll observe that I have an application that streams 4 x 10Msps in the
*receive* direction from an X310, and I don't experience
'O' at all. The application is sometimes used to stream for 24 or
more hours at a time. Now, this isn't an N310, but the N310
shares a LOT of FPGA architecture with the X310, and the streaming is
in the other direction. So, just an anecdote that is only
very-slightly useful.
On Thu, Apr 13, 2023 at 5:03 PM Jim Schatzman
<james.schatz...@futurelabusa.com> wrote:
Even with all the recommended settings, and a very fast computer
that is doing nothing except sending the data, it is maybe 50/50
that a 2 hour simulation can be conducted without an underrun. The
longest run I have been able to do without an underrun is about
2.5 hours.
The sample rate is 12.5 Msamp/sec at 16 bit I + 16 bit Q or 400
Mbit/sec.
For our application, that is unacceptable. I need to be able to
run for days without data loss.
It is a mystery to me why a 10 Gbit connection cannot support 400
Mbit/sec UDP reliably.
Any ideas about how we can completely eliminate underruns?
At the moment, I am uncertain whether the problem is occurring on
the host or on the radio. I suspect the radio, but I will do some
testing of the host to see what UDP data rate it can support
without loss.
Thanks!
At 10:53 PM 4/10/2023, Marcus D. Leech wrote:
>On 10/04/2023 21:12, Jim Schatzman wrote:
>>The following steps had no impact:
>>
>>a) Don't use a switch but do a point-to-point connection between
the comptuer's NIC and the N310.
>>b) Adjust network buffers and ring buffer per recommendations
(actually, the network buffers setting was always being done).
>>
>>Increasing the MTU to 9000 had a significant impact. An
occasionaly underrun is still experienced, but an hour or two of
continuous transmission is possible.
>>
>>I am wondering if this is not an issue on the computer side but
on the radio side. Is the N310 incapable of supporting 1x 10 Gbps
streams with a MTU of 1500? Perhaps.
>>
>>I will do some computer-to-computer testing to see if the
problem can be reproduced there.
>>
>>Jim
>Even non-SDR applications tend to use jumbo-frames for continuous
traffic at 10Gbit. I'm sorry that I didn't clue in to that
>Â in the first round.
>
>
>>
>>
>>
>>
>>At 08:39 PM 4/7/2023, Marcus D. Leech wrote:
>>>On 07/04/2023 22:13, Jim Schatzman wrote:
>>>>We have been unable to estable 100% reliable connections to an
N310 USRP radio through its 10 Gbit ethernet from Linux. What
happens is that it works fine for a period of time - 30 to 60
minutes, typically. Then we see a couple of U's in the output.
Unfortunately, that is fatal for our application.
>>>>
>>>>Using the unmodified tx_samples_from_file or one modified to
use separate threads to read data from the file and to sent it
over the socket to the radio, the symptoms are the same.
>>>>
>>>>All the evidence is that the application is sending data
continuously without any delays. Also, the "network" has no
devices on it except for the host computer, a high performance 10G
switch, and the N310 radio.
>>>>
>>>>We are wondering if this could be a Linux "feature". We would
like to try increasing the socket priority with SO_PRIORITY.
However, we are not finding any hooks in the UHD software for this.
>>>>
>>>>Suggestions?
>>>>
>>>>Thanks!
>>>>Jim
>>>>_______________________________________________
>>>Have you increased the ring buffers on your card?
>>>
>>>https://kb.ettus.com/USRP_Host_Performance_Tuning_Tips_and_Tricks#Increasing_Ring_Buffers
>>>
>>>Also, adjust the network buffers:
>>>
>>>https://kb.ettus.com/USRP_Host_Performance_Tuning_Tips_and_Tricks#Adjust_Network_Buffers
>>>
>>>Have you tried a direct connection--without the switch? Â
 Just to eliminate it. Before you say "oh, it's not the switch",
>>>Â many "high performance" 10G switches have a less-than-ideal
switching fabric.
>>>
>>>_______________________________________________ USRP-users
mailing list -- usrp-users@lists.ettus.com To unsubscribe send an
email to usrp-users-le...@lists.ettus.com </x-flowed>
>>_______________________________________________
>>USRP-users mailing list -- usrp-users@lists.ettus.com
>>To unsubscribe send an email to usrp-users-le...@lists.ettus.com
>_______________________________________________ USRP-users
mailing list -- usrp-users@lists.ettus.com To unsubscribe send an
email to usrp-users-le...@lists.ettus.com </x-flowed>
_______________________________________________
USRP-users mailing list -- usrp-users@lists.ettus.com
To unsubscribe send an email to usrp-users-le...@lists.ettus.com
_______________________________________________
USRP-users mailing list -- usrp-users@lists.ettus.com
To unsubscribe send an email to usrp-users-le...@lists.ettus.com
