Re: Reducing rare underflows to zero occurrences

[Chad Spooner]

Daniel:

Thanks much for the tips.



   ss -ulm

and recorded the results.



   State  Recv-Q Send-Q Local Address:Port   Peer Address:PortProcess

   UNCONN 0      0      127.0.0.53%lo:domain      0.0.0.0:*
         skmem:(r0,rb212992,t0,tb212992,f4096,w0,o0,bl0,d0)
   UNCONN 0      0            0.0.0.0:49206       0.0.0.0:*
         skmem:(r0,rb212992,t0,tb212992,f0,w0,o0,bl0,d0)
   UNCONN 0      0            0.0.0.0:631         0.0.0.0:*
         skmem:(r0,rb212992,t0,tb212992,f0,w0,o0,bl0,d0)
   UNCONN 0      0            0.0.0.0:mdns        0.0.0.0:*
         skmem:(r0,rb212992,t0,tb212992,f4096,w0,o144,bl0,d0)



   Uusrp_sink :error: In the last 8307 ms, 1 underflows occurred.
   usrp_sink :error: In the last 340937 ms, 1 underflows occurred.
   Uusrp_sink :error: In the last 171004 ms, 1 underflows occurred.
   Uusrp_sink :error: In the last 127998 ms, 1 underflows occurred.
   UUusrp_sink :error: In the last 640023 ms, 1 underflows occurred.
   usrp_sink :error: In the last 511982 ms, 1 underflows occurred.
   UUusrp_sink :error: In the last 256048 ms, 1 underflows occurred.
   usrp_sink :error: In the last 127946 ms, 1 underflows occurred.
   Uusrp_sink :error: In the last 1963006 ms, 1 underflows occurred.
   Uusrp_sink :error: In the last 128000 ms, 1 underflows occurred.
   UUusrp_sink :error: In the last 384001 ms, 1 underflows occurred.
   usrp_sink :error: In the last 171006 ms, 1 underflows occurred.
   UUusrp_sink :error: In the last 85083 ms, 1 underflows occurred.
   usrp_sink :error: In the last 42916 ms, 1 underflows occurred.
   UUusrp_sink :error: In the last 85082 ms, 1 underflows occurred.
   Uusrp_sink :error: In the last 127925 ms, 1 underflows occurred.
   Uusrp_sink :error: In the last 43042 ms, 1 underflows occurred.
   usrp_sink :error: In the last 127950 ms, 1 underflows occurred.
   Uusrp_sink :error: In the last 128000 ms, 1 underflows occurred.
   Uusrp_sink :error: In the last 127998 ms, 1 underflows occurred.
   Uusrp_sink :error: In the last 127999 ms, 1 underflows occurred.
   Uusrp_sink :error: In the last 128000 ms, 1 underflows occurred.
   Uusrp_sink :error: In the last 127999 ms, 1 underflows occurred.
   Uusrp_sink :error: In the last 384001 ms, 1 underflows occurred.
   UUusrp_sink :error: In the last 127999 ms, 1 underflows occurred.
   usrp_sink :error: In the last 128000 ms, 1 underflows occurred.
   UUusrp_sink :error: In the last 256082 ms, 1 underflows occurred.
   Uusrp_sink :error: In the last 127940 ms, 1 underflows occurred.
   Uusrp_sink :error: In the last 427075 ms, 1 underflows occurred.
   Uusrp_sink :error: In the last 127941 ms, 1 underflows occurred.
   Uusrp_sink :error: In the last 384023 ms, 1 underflows occurred.
   usrp_sink :error: In the last 127942 ms, 1 underflows occurred.
   Uusrp_sink :error: In the last 128000 ms, 1 underflows occurred.
   UUusrp_sink :error: In the last 384023 ms, 1 underflows occurred.
   usrp_sink :error: In the last 255977 ms, 1 underflows occurred.
   Uusrp_sink :error: In the last 511999 ms, 1 underflows occurred.
   UUusrp_sink :error: In the last 384024 ms, 1 underflows occurred.

which is weird and probably significant?

I hope my use of ss was correct.

Thanks again,

Chad

On 1/16/25 1:49 PM, Daniel Estévez wrote:

I tried to monitor /proc/net/udp while the channel flowgraph was running over a couple hours. Every second I called I never saw Recv-Q or Send-Q deviate from 0, and all occurrences of the dropped-packed indicator were also 0: During this multi-hour experiment, there were several reported underflows, and I noticed that many of them are separated by 128 seconds, 512 seconds, 256 seconds, and combinations like 384 = 128 + 256 seconds:

Hi Chad,



Best,
Daniel.

On 16/01/2025 19:38, Chad Spooner wrote:

Something that you can do to monitor the problem is to look in /proc/net/udp. This will show you the fill level of kernel UDP receive buffers. If you've set the buffer size to 32 MiB as recommended by the performance tuning tips, usually these sporadic drops are the result of the flowgraph consuming data at a rate slightly lower than the nominal sample rate for multiple seconds, rather than a brief but larger performance reduction. If this is the case, you can see how the buffer fill level slowly increases over the course of several seconds until the buffer fills up and packets are dropped. Looking at this won't solve your problem, but at least it will give you a way to monitor if you do some changes to try to improve the performance.

All:












    [INFO] [X300] X300 initialization sequence...
    [INFO] [X300] Maximum frame size: 8000 bytes.
    [INFO] [X300] Radio 1x clock: 200 MHz
    usrp_source :info: set_min_output_buffer on block 1 to 2097152
    usrp_sink :info: set_min_output_buffer on block 2 to 2097152
    mmse_resampler_cc :info: set_min_output_buffer on block 4 to 2097152
    fir_filter_blk<IN_T,OUT_T,TAP_T> :info: set_min_output_buffer on
    block 5 to 2097152
    multiply_cc :info: set_min_output_buffer on block 9 to 2097152
    sig_source :info: set_min_output_buffer on block 8 to 2097152
    add_cc :info: set_min_output_buffer on block 6 to 2097152
    fastnoise_source :info: set_min_output_buffer on block 7 to 2097152
    *Uusrp_sink :error: In the last 579300 ms, 1 underflows occurred.
    Uusrp_sink :error: In the last 256008 ms, 1 underflows occurred.
    usrp_sink :error: In the last 84992 ms, 1 underflows occurred.
    UUusrp_sink :error: In the last 128008 ms, 1 underflows occurred.*



System Information:

    24-core Intel i9 system
    Ubuntu 22.04.5 LTS, kernel 5.19.0-46-generic #47~22.04.1-Ubuntu
    gnuradio 3.10.1.1
    UHD 4.1.0.5-3



https://kb.ettus.com/USRP_Host_Performance_Tuning_Tips_and_Tricks

I've also tinkered with the minoutbuf parameter in the various blocks.

Any advice?

Thanks,

Chad
cyclostationary.blog
cmspoo...@nwra.com





--
Chad M. Spooner, PhD
NorthWest Research Associates
149 Bonifacio Place, Suite 200
Monterey, CA 93940
cmspoo...@nwra.com
831 521 6743
NWRA Scientist/Engineer, VP, BoD
cyclostationary.blog

I'm developing a comm system involving three SDRs: transmit, channel effects, and receiver. The channel SDR is controlled by a flowgraph that has various options for the channel: AWGN (h(\tau) = \delta(\tau)), LTI, and a couple time- variant frequency-selective channel models. I need to run the full three-SDR system over long periods of time to get stable average values for system-performance parameters. My problem is that the channel flowgraph reports underflows on the order of every hundred seconds or so (sometimes much less). I've tracked down these rare underflows in the channel to transient problems in the receiver flowgraph. So I put together a minimum flowgraph that shows the problem, which is attached. When I run it, I get output like this:     [INFO] [UHD] linux; GNU C++ version 11.2.0; Boost_107400; UHD_4.1.0.5-3 The PSD of the filtered input signal appears correctly on a spectrum analyzer. I've taken some pains to optimize the OS settings for gnuradio/UHD along the lines suggested by the online tips at

--
Chad M. Spooner, PhD
NorthWest Research Associates
149 Bonifacio Place, Suite 200
Monterey, CA 93940
cmspoo...@nwra.com
831 521 6743
NWRA Scientist/Engineer, VP, BoD
cyclostationary.blog

smime.p7s
Description: S/MIME Cryptographic Signature