Monitor O's and U's from Python

[Richard Bell]

Hello,

I have a python flowgraph that transmits data from one radio to another for
a research project. Sometimes during one of these tx/rx events, U's or O's
will occur, making the data invalid. I would like to monitor the USRP for
these events from within Python so I can request a retransmit when they
occur.

How do I monitor for O's and U's from within Python?

Thank you,

Richard

Re: Monitor O's and U's from Python

[Marcus Müller]

Hello Richard,

For RX / "O":

you can observe the RX timestamps (key: rx_time) in tags, and compare with the last tag 
you saw, to check whether the difference of `rx_time` value times the sampling rate is the 
same as the difference in tag offset (i.e., sample number on which the tag was).


For both TX / "U" and RX / "O":

The USRP source and sinkl blocks have an "async_msg" message output , which will give you 
message dicts that (pseudocoding this):


{ "uhd_async_msg" { "channel" : 0 , "time_spec" : [0, 0.0713], "event_code": 
"underflow" } }


Best,

Marcus

On 21.04.23 09:08, Richard Bell wrote:


Hello,

I have a python flowgraph that transmits data from one radio to another for a research 
project. Sometimes during one of these tx/rx events, U's or O's will occur, making the 
data invalid. I would like to monitor the USRP for these events from within Python so I 
can request a retransmit when they occur.


How do I monitor for O's and U's from within Python?

Thank you,

Richard

Re: libgnuradio-runtime problem when capturing GNSS signals with HackRF and gnss-sdr

[Marcus Müller]

Hi,
the GR version you report to have installed (3.10.1) and the version of 
libgnuradio-runtime (3.10.6git) differ: I think you're trying to mix different versions of 
GNU Radio, and especially, try to load a library (libgnuradio-osmosdr.so) that was linked 
against a different version of GNU Radio with GNU Radio.


That won't work. I'm not an expert in GNSS-SDR, but in all honesty, when methodology 
requires you LD_PRELOAD a single library, you're always trying to "bend" something around; 
and unless you really meant to replace one library with another that does something 
different, that's *usually* not what you want.


So:

> We don't know what could be the problem,

Almost certainly that you are using LD_PRELOAD with a different version of GNU 
Radio.

Best regards,
Marcus

On 21.04.23 15:38, Katarina Radoš wrote:

Hi everyone,

we are trying to capture GNSS signals using HackRF and gnss-sdr. We have installed 
gnss-sdr, gnuradio (version 3.10.1) and gr-osmosdr (version 0.2.3). When we try to run the 
following command $ 
LD_PRELOAD=/home/gnss/gnss-sdr/gr-osmosdr/build/lib/libgnuradio-osmosdr.so 
/home/gnss/gnss-sdr/build/src/main/gnss-sdr -c hackrf_GPS_L1.conf, we get the following 
error /home/gnss/gnss-sdr/build/src/main/gnss-sdr: error while loading shared libraries: 
libgnuradio-runtime.so.3.10.6git: cannot open shared object file: No such file or directory.


We don't know what could be the problem, maybe someone had the same problem? Thanks in 
advance for your help.


Best regards,
KR

Re: double free or corruption error when two instances of an OOT block using fftw is called in GRC

[Marcus Müller]

Hi Grace,

reading this backtrace, the error happens in

#29 0x7fffe99b04d5 in gr::customModule::probeExtract_impl::ApplyFilterBank

So, that's where you need to look, not sure whether the other code you shared 
helps.

But reading your other code: you seem to be making an FFT plan and destroying it *every 
single time you do an FFT*. That's a terrible idea – planning an FFT typically takes much 
longer than doing an FFT, even a really badly implemented one.

You also make a local array of plans, just to put that single plan in, then 
destroy it.
This indicates you've got a logical bug in your thinking about C++ scoping and lifetimes. 
If I had to take a wild guess, you're trying to access a plan that does no longer exist in 
your ApplyFilterBank.


Best regards,
Marcus

On 19.04.23 20:18, Grace Yeung wrote:
After some debugging, it appears grc is breaking inside of calls to fftw (see below). I am 
simply trying to use fftw to perform a simple FFT transform (forward/backward). Is there 
anything special I need to watch out for when using fftw with gnuradio? Any examples 
available? Any advice would be much appreciated. Thanks!


Backtracing shows the crash happens in an fftw3f library function used in the 
code...

(gdb) bt
#0 __GI_raise (sig=sig@entry=6) at ../sysdeps/unix/sysv/linux/raise.c:50
#1 0x77dd1859 in __GI_abort () at 
abort.c:79
#2 0x77e3c26e in __libc_message 
(action=action@entry=do_abort, fmt=fmt@entry=0x77f66298 "%s\n") at 
../sysdeps/posix/libc_fatal.c:155
#3 0x77e442fc in malloc_printerr 
(str=str@entry=0x77f68628 "double free or corruption (fasttop)") at malloc.c:5347
#4 0x77e45c65 in _int_free 
(av=0x7fffac20, p=0x7fffac55a8d0, have_lock=0) at malloc.c:4266
#5 0x7fffefe830d9 in ?? () from 
/lib/x86_64-linux-gnu/libfftw3f.so.3
#6 0x7fffefe83c44 in ?? () from 
/lib/x86_64-linux-gnu/libfftw3f.so.3
#7 0x7fffefe83fc4 in fftwf_mkplan_d () 
from /lib/x86_64-linux-gnu/libfftw3f.so.3
#8 0x7fffefe88e5d in ?? () from 
/lib/x86_64-linux-gnu/libfftw3f.so.3
#9 0x7fffefe8380b in ?? () from 
/lib/x86_64-linux-gnu/libfftw3f.so.3
#10 0x7fffefe83a03 in ?? () from 
/lib/x86_64-linux-gnu/libfftw3f.so.3
#11 0x7fffefe83fc4 in fftwf_mkplan_d () 
from /lib/x86_64-linux-gnu/libfftw3f.so.3
#12 0x7fffefe8d95c in ?? () from 
/lib/x86_64-linux-gnu/libfftw3f.so.3
#13 0x7fffefe8380b in ?? () from 
/lib/x86_64-linux-gnu/libfftw3f.so.3
#14 0x7fffefe83a03 in ?? () from 
/lib/x86_64-linux-gnu/libfftw3f.so.3
#15 0x7fffefe83fc4 in fftwf_mkplan_d () 
from /lib/x86_64-linux-gnu/libfftw3f.so.3
#16 0x7fffefe84043 in fftwf_mkplan_f_d 
() from /lib/x86_64-linux-gnu/libfftw3f.so.3
#17 0x7fffefe887a3 in ?? () from 
/lib/x86_64-linux-gnu/libfftw3f.so.3
#18 0x7fffefe8380b in ?? () from 
/lib/x86_64-linux-gnu/libfftw3f.so.3
#19 0x7fffefe83a03 in ?? () from 
/lib/x86_64-linux-gnu/libfftw3f.so.3
#20 0x7fffefe83fc4 in fftwf_mkplan_d () 
from /lib/x86_64-linux-gnu/libfftw3f.so.3
#21 0x7fffefe88e5d in ?? () from 
/lib/x86_64-linux-gnu/libfftw3f.so.3
#22 0x7fffefe8380b in ?? () from 
/lib/x86_64-linux-gnu/libfftw3f.so.3
#23 0x7fffefe83a03 in ?? () from 
/lib/x86_64-linux-gnu/libfftw3f.so.3
#24 0x7fffeff4a583 in ?? () from 
/lib/x86_64-linux-gnu/libfftw3f.so.3
#25 0x7fffeff4a758 in fftwf_mkapiplan () 
from /lib/x86_64-linux-gnu/libfftw3f.so.3
#26 0x7fffeff4ddd7 in 
fftwf_plan_many_dft () from /lib/x86_64-linux-gnu/libfftw3f.so.3
#27 0x7fffeff4d09b in fftwf_plan_dft () 
from /lib/x86_64-linux-gnu/libfftw3f.so.3
#28 0x7fffeff4cd9a in fftwf_plan_dft_1d 
() from /lib/x86_64-linux-gnu/libfftw3f.so.3
#29 0

Re: double free or corruption error when two instances of an OOT block using fftw is called in GRC

[Grace Yeung]

Hi Marcus,

Here is the ForwardFT() code below that is called by ApplyFilterBank() 
in the work function. So does it mean I should compute the plan only 
once in the constructor and use it throughout the work function by 
passing it to the ForwardFT() funciton? And declare _plans in the impl.h 
file?  Thanks.


Grace

  void probeExtract_impl::ForwardFT (vector &series, 
int M)

  {
  int exponent;
  fftwf_plan plans[MAX_FFT_EXP];

  exponent = (int) (log10((float) M)/log10(2.0) + 0.5);

  err_val = fftwf_init_threads ();
  if (err_val == 0)
  throw runtime_error("fftwf_init_threads failed");
  fftwf_plan_with_nthreads(NUM_FFTW_THREADS);

  pthread_mutex_lock (&fftw3_mutex);

  fftwf_make_planner_thread_safe();

  plans[exponent] = fftwf_plan_dft_1d (M,
reinterpret_cast (&series[0]),
reinterpret_cast (&series[0]),
   FFTW_FORWARD, 
FFTW_ESTIMATE);

  fftwf_execute (plans[exponent]);

  pthread_mutex_unlock (&fftw3_mutex);
  fftwf_destroy_plan (plans[exponent]);
  fftwf_cleanup_threads ();
  // Notes: destroying plans/cleaning up threads appear to 
eliminate growing memory issue 2/2023


  return;
  }

On 4/21/23 07:11, Marcus Müller wrote:

Hi Grace,

reading this backtrace, the error happens in

#29 0x7fffe99b04d5 in 
gr::customModule::probeExtract_impl::ApplyFilterBank


So, that's where you need to look, not sure whether the other code you 
shared helps.


But reading your other code: you seem to be making an FFT plan and 
destroying it *every single time you do an FFT*. That's a terrible 
idea – planning an FFT typically takes much longer than doing an FFT, 
even a really badly implemented one.
You also make a local array of plans, just to put that single plan in, 
then destroy it.
This indicates you've got a logical bug in your thinking about C++ 
scoping and lifetimes. If I had to take a wild guess, you're trying to 
access a plan that does no longer exist in your ApplyFilterBank.


Best regards,
Marcus

On 19.04.23 20:18, Grace Yeung wrote:
After some debugging, it appears grc is breaking inside of calls to 
fftw (see below). I am simply trying to use fftw to perform a simple 
FFT transform (forward/backward). Is there anything special I need to 
watch out for when using fftw with gnuradio? Any examples available? 
Any advice would be much appreciated. Thanks!


Backtracing shows the crash happens in an fftw3f library function 
used in the code...


(gdb) bt
#0 __GI_raise (sig=sig@entry=6) at ../sysdeps/unix/sysv/linux/raise.c:50
#1 0x77dd1859 in 
__GI_abort () at abort.c:79
#2 0x77e3c26e in 
__libc_message (action=action@entry=do_abort, 
fmt=fmt@entry=0x77f66298 "%s\n") at 
../sysdeps/posix/libc_fatal.c:155
#3 0x77e442fc in 
malloc_printerr (str=str@entry=0x77f68628 "double free or 
corruption (fasttop)") at malloc.c:5347
#4 0x77e45c65 in 
_int_free (av=0x7fffac20, p=0x7fffac55a8d0, have_lock=0) at 
malloc.c:4266
#5 0x7fffefe830d9 in 
?? () from /lib/x86_64-linux-gnu/libfftw3f.so.3
#6 0x7fffefe83c44 in 
?? () from /lib/x86_64-linux-gnu/libfftw3f.so.3
#7 0x7fffefe83fc4 in 
fftwf_mkplan_d () from /lib/x86_64-linux-gnu/libfftw3f.so.3
#8 0x7fffefe88e5d in 
?? () from /lib/x86_64-linux-gnu/libfftw3f.so.3
#9 0x7fffefe8380b in 
?? () from /lib/x86_64-linux-gnu/libfftw3f.so.3
#10 0x7fffefe83a03 
in ?? () from /lib/x86_64-linux-gnu/libfftw3f.so.3
#11 0x7fffefe83fc4 
in fftwf_mkplan_d () from /lib/x86_64-linux-gnu/libfftw3f.so.3
#12 0x7fffefe8d95c 
in ?? () from /lib/x86_64-linux-gnu/libfftw3f.so.3
#13 0x7fffefe8380b 
in ?? () from /lib/x86_64-linux-gnu/libfftw3f.so.3
#14 0x7fffefe83a03 
in ?? () from /lib/x86_64-linux-gnu/libfftw3f.so.3
#15 0x7fffefe83fc4 
in fftwf_mkplan_d () from /lib/x86_64-linux-gnu/libfftw3f.so.3
#16 0x7fffefe84043 
in fftwf_mkplan_f_d () from /lib/x86_64-linux-gnu/libfftw3f.so.3
#17 0x7fffefe887a3 
in ?? () from /lib/x86_64-linux-gnu/libfftw3f.so.3
#18 0x7fffefe8380b 
in ?? () from /lib/x86_64-

Re: double free or corruption error when two instances of an OOT block using fftw is called in GRC

[Grace Yeung]

Thank you Marcus for taking the time to look through the backtrace 
error. ApplyFilterBank simply calls ForwardFT to perform the FFT and 
does not access the plan. It only uses the FFT output. I had to destroy 
the plan each time because I noticed memory was consistently growing 
when gnuradio was running the flowgraph. After adding the destroy_plan 
memory stops growing. Hope this makes sense. Thank you.


Grace



On 4/21/23 07:11, Marcus Müller wrote:

Hi Grace,

reading this backtrace, the error happens in

#29 0x7fffe99b04d5 in 
gr::customModule::probeExtract_impl::ApplyFilterBank


So, that's where you need to look, not sure whether the other code you 
shared helps.


But reading your other code: you seem to be making an FFT plan and 
destroying it *every single time you do an FFT*. That's a terrible 
idea – planning an FFT typically takes much longer than doing an FFT, 
even a really badly implemented one.
You also make a local array of plans, just to put that single plan in, 
then destroy it.
This indicates you've got a logical bug in your thinking about C++ 
scoping and lifetimes. If I had to take a wild guess, you're trying to 
access a plan that does no longer exist in your ApplyFilterBank.


Best regards,
Marcus

On 19.04.23 20:18, Grace Yeung wrote:
After some debugging, it appears grc is breaking inside of calls to 
fftw (see below). I am simply trying to use fftw to perform a simple 
FFT transform (forward/backward). Is there anything special I need to 
watch out for when using fftw with gnuradio? Any examples available? 
Any advice would be much appreciated. Thanks!


Backtracing shows the crash happens in an fftw3f library function 
used in the code...


(gdb) bt
#0 __GI_raise (sig=sig@entry=6) at ../sysdeps/unix/sysv/linux/raise.c:50
#1 0x77dd1859 in 
__GI_abort () at abort.c:79
#2 0x77e3c26e in 
__libc_message (action=action@entry=do_abort, 
fmt=fmt@entry=0x77f66298 "%s\n") at 
../sysdeps/posix/libc_fatal.c:155
#3 0x77e442fc in 
malloc_printerr (str=str@entry=0x77f68628 "double free or 
corruption (fasttop)") at malloc.c:5347
#4 0x77e45c65 in 
_int_free (av=0x7fffac20, p=0x7fffac55a8d0, have_lock=0) at 
malloc.c:4266
#5 0x7fffefe830d9 in 
?? () from /lib/x86_64-linux-gnu/libfftw3f.so.3
#6 0x7fffefe83c44 in 
?? () from /lib/x86_64-linux-gnu/libfftw3f.so.3
#7 0x7fffefe83fc4 in 
fftwf_mkplan_d () from /lib/x86_64-linux-gnu/libfftw3f.so.3
#8 0x7fffefe88e5d in 
?? () from /lib/x86_64-linux-gnu/libfftw3f.so.3
#9 0x7fffefe8380b in 
?? () from /lib/x86_64-linux-gnu/libfftw3f.so.3
#10 0x7fffefe83a03 
in ?? () from /lib/x86_64-linux-gnu/libfftw3f.so.3
#11 0x7fffefe83fc4 
in fftwf_mkplan_d () from /lib/x86_64-linux-gnu/libfftw3f.so.3
#12 0x7fffefe8d95c 
in ?? () from /lib/x86_64-linux-gnu/libfftw3f.so.3
#13 0x7fffefe8380b 
in ?? () from /lib/x86_64-linux-gnu/libfftw3f.so.3
#14 0x7fffefe83a03 
in ?? () from /lib/x86_64-linux-gnu/libfftw3f.so.3
#15 0x7fffefe83fc4 
in fftwf_mkplan_d () from /lib/x86_64-linux-gnu/libfftw3f.so.3
#16 0x7fffefe84043 
in fftwf_mkplan_f_d () from /lib/x86_64-linux-gnu/libfftw3f.so.3
#17 0x7fffefe887a3 
in ?? () from /lib/x86_64-linux-gnu/libfftw3f.so.3
#18 0x7fffefe8380b 
in ?? () from /lib/x86_64-linux-gnu/libfftw3f.so.3
#19 0x7fffefe83a03 
in ?? () from /lib/x86_64-linux-gnu/libfftw3f.so.3
#20 0x7fffefe83fc4 
in fftwf_mkplan_d () from /lib/x86_64-linux-gnu/libfftw3f.so.3
#21 0x7fffefe88e5d 
in ?? () from /lib/x86_64-linux-gnu/libfftw3f.so.3
#22 0x7fffefe8380b 
in ?? () from /lib/x86_64-linux-gnu/libfftw3f.so.3
#23 0x7fffefe83a03 
in ?? () from /lib/x86_64-linux-gnu/libfftw3f.so.3
#24 0x7fffeff4a583 
in ?? () from /lib/x86_64-linux-gnu/libfftw3f.so.3
#25 0x7fffeff4a758 
in fftwf_mkapiplan () from /lib/x86_64-linux-gnu/libfft

Re: double free or corruption error when two instances of an OOT block using fftw is called in GRC

[Marcus Müller]

You leaking memory if you don't destroy plans is not surprising. It still makes no sense 
how you make an array, just to throw it away, as explained in my previous email. Your code 
really is not sensible, sorry!


Best regards,
Marcus

On 21.04.23 17:20, Grace Yeung wrote:
Thank you Marcus for taking the time to look through the backtrace error. ApplyFilterBank 
simply calls ForwardFT to perform the FFT and does not access the plan. It only uses the 
FFT output. I had to destroy the plan each time because I noticed memory was consistently 
growing when gnuradio was running the flowgraph. After adding the destroy_plan memory 
stops growing. Hope this makes sense. Thank you.


Grace

—​
Grace K. Yeung, MS
NorthWest Research Associates
301 Webster Street
Monterey, CA 93940
gr...@nwra.com
https://www.nwra.com
--
*From:* Marcus Müller 
*Sent:* Friday, April 21, 2023 7:10:43 AM
*To:* Grace Yeung 
*Subject:* Re: double free or corruption error when two instances of an OOT block using 
fftw is called in GRC

Hi Grace,

reading this backtrace, the error happens in

#29 0x7fffe99b04d5 in gr::customModule::probeExtract_impl::ApplyFilterBank

So, that's where you need to look, not sure whether the other code you shared 
helps.

But reading your other code: you seem to be making an FFT plan and destroying 
it *every
single time you do an FFT*. That's a terrible idea – planning an FFT typically 
takes much
longer than doing an FFT, even a really badly implemented one.
You also make a local array of plans, just to put that single plan in, then 
destroy it.
This indicates you've got a logical bug in your thinking about C++ scoping and 
lifetimes.
If I had to take a wild guess, you're trying to access a plan that does no 
longer exist in
your ApplyFilterBank.

Best regards,
Marcus

On 19.04.23 20:18, Grace Yeung wrote:

After some debugging, it appears grc is breaking inside of calls to fftw (see 
below). I am
simply trying to use fftw to perform a simple FFT transform (forward/backward). 
Is there
anything special I need to watch out for when using fftw with gnuradio? Any 
examples
available? Any advice would be much appreciated. Thanks!

Backtracing shows the crash happens in an fftw3f library function used in the 
code...

(gdb) bt
#0 __GI_raise (sig=sig@entry=6) at ../sysdeps/unix/sysv/linux/raise.c:50
#1 >0x77dd1859 in __GI_abort 
() at

abort.c:79
#2 >0x77e3c26e in 
__libc_message
(action=action@entry=do_abort, fmt=fmt@entry=0x77f66298 "%s\n") at 
../sysdeps/posix/libc_fatal.c:155
#3 >0x77e442fc in 
malloc_printerr

(str=str@entry=0x77f68628 "double free or corruption (fasttop)") at 
malloc.c:5347
#4 >0x77e45c65 in _int_free

(av=0x7fffac20, p=0x7fffac55a8d0, have_lock=0) at malloc.c:4266
#5 >0x7fffefe830d9 in ?? () from

/lib/x86_64-linux-gnu/libfftw3f.so.3
#6 >0x7fffefe83c44 in ?? () from

/lib/x86_64-linux-gnu/libfftw3f.so.3
#7 >0x7fffefe83fc4 in 
fftwf_mkplan_d ()

from /lib/x86_64-linux-gnu/libfftw3f.so.3
#8 >0x7fffefe88e5d in ?? () from

/lib/x86_64-linux-gnu/libfftw3f.so.3
#9 >0x7fffefe8380b in ?? () from

/lib/x86_64-linux-gnu/libfftw3f.so.3
#10 >0x7fffefe83a03 in ?? () from

/lib/x86_64-linux-gnu/libfftw3f.so.3
#11 >0x7fffefe83fc4 in 
fftwf_mkplan_d ()

from /lib/x86_64-linux-gnu/libfftw3f.so.3
#12 >0x7fffefe8d95c in ?? () from

/lib/x86_64-linux-gnu/libfftw3f.so.3
#13 >0x7fffefe8380b in ?? () from

/lib/x86_64-linux-gnu/libfftw3f.so.3
#14 >0x7fffefe83a03 in ?? () from

/lib/x86_64-linux-gnu/libfftw3f.so.3
#15 

Re: double free or corruption error when two instances of an OOT block using fftw is called in GRC

[Marcus Müller]

Hi Grace,

as said, this code shows that you don't understand how long (or rather: short) the object 
`plans` lives! This is a basic C++ issue: things created in local scope of a function end 
to exist at the end of the function. (I honestly think you'll need a bit of a better 
introduction to C++ – this will be easier for you with a solid understanding!)


So, your code needs fundamental rework:

- You currently always have to make a new FFTW plan, because your input might always be at 
a new address. That's bad. Instead, you must always put the samples into the same input 
buffer of your FFT. Then you can reuse the same plan over and over again.
- That means you have to rewrite ApplyFilterBank, so that it always puts the samples into 
the input buffer of your FFT.
- As soon as you've done that, you can use `gr::fft::fft` (which gives you an 
FFTw of type float that is forward=true), instead of doing **anything** with FFTW yourself.

- You can then just throw away your `ForwardFT` function. Instead,
  - add a `std::unordered_map> 
prepared_ffts` to
your class's private members in probeExtract_impl.h, then
  - in `ApplyFilterBank`, you just do something like

auto& [fft, already_there] = prepared_ffts.try_emplace(M, M, 
number_of_threads);
auto& input_buffer = fft.get_inbuf();
// TODO: put all input into input_buffer
fft.execute()
// output is now in fft.get_outbuf()

That's it. You don't have to care about any fftwf_init_threads, about any _delete_plan, 
you don't need to create the plans yourself.


Best,
Marcus

On 21.04.23 17:53, Grace Yeung wrote:


Hi Marcus,

Here is the ForwardFT() code below that is called by ApplyFilterBank() in the work 
function. So does it mean I should compute the plan only once in the constructor and use 
it throughout the work function by passing it to the ForwardFT() funciton? And declare 
_plans in the impl.h file?  Thanks.


Grace

   void probeExtract_impl::ForwardFT (vector &series, int M)
   {
   int exponent;
   fftwf_plan plans[MAX_FFT_EXP];

   exponent = (int) (log10((float) M)/log10(2.0) + 0.5);

   err_val = fftwf_init_threads ();
   if (err_val == 0)
   throw runtime_error("fftwf_init_threads failed");
   fftwf_plan_with_nthreads(NUM_FFTW_THREADS);

   pthread_mutex_lock (&fftw3_mutex);

   fftwf_make_planner_thread_safe();

   plans[exponent] = fftwf_plan_dft_1d (M,
reinterpret_cast (&series[0]),
reinterpret_cast (&series[0]),
    FFTW_FORWARD, FFTW_ESTIMATE);
   fftwf_execute (plans[exponent]);

   pthread_mutex_unlock (&fftw3_mutex);
   fftwf_destroy_plan (plans[exponent]);
   fftwf_cleanup_threads ();
   // Notes: destroying plans/cleaning up threads appear to eliminate growing 
memory issue 2/2023


   return;
   }


On 4/21/23 08:20, Grace Yeung wrote:
Thank you Marcus for taking the time to look through the backtrace error. 
ApplyFilterBank simply calls ForwardFT to perform the FFT and does not access the plan. 
It only uses the FFT output. I had to destroy the plan each time because I noticed 
memory was consistently growing when gnuradio was running the flowgraph. After adding 
the destroy_plan memory stops growing. Hope this makes sense. Thank you.


Grace

—​
Grace K. Yeung, MS
NorthWest Research Associates
301 Webster Street
Monterey, CA 93940
gr...@nwra.com
https://www.nwra.com
--
*From:* Marcus Müller 
*Sent:* Friday, April 21, 2023 7:10:43 AM
*To:* Grace Yeung 
*Subject:* Re: double free or corruption error when two instances of an OOT block using 
fftw is called in GRC

Hi Grace,

reading this backtrace, the error happens in

#29 0x7fffe99b04d5 in gr::customModule::probeExtract_impl::ApplyFilterBank

So, that's where you need to look, not sure whether the other code you shared 
helps.

But reading your other code: you seem to be making an FFT plan and destroying 
it *every
single time you do an FFT*. That's a terrible idea – planning an FFT typically 
takes much
longer than doing an FFT, even a really badly implemented one.
You also make a local array of plans, just to put that single plan in, then 
destroy it.
This indicates you've got a logical bug in your thinking about C++ scoping and 
lifetimes.
If I had to take a wild guess, you're trying to access a plan that does no 
longer exist in
your ApplyFilterBank.

Best regards,
Marcus

On 19.04.23 20:18, Grace Yeung wrote:
> After some debugging, it appears grc is breaking inside of calls to fftw (see below). 
I am

> simply trying to use fftw to perform a simple FFT transform 
(forward/backward). Is there
> anything special I need to watch out for when using fftw with gnuradio? Any 
examples
> available? Any advice would be much appreciated. Thanks!
>
> Backtracing shows the crash happens in an f

Re: double free or corruption error when two instances of an OOT block using fftw is called in GRC

[Grace Yeung]

Thank you Marcus. Yes, the double free error is gone now that I create 
the plan just once in the beginning. Thank you for your patience.


Grace

On 4/21/23 12:30, Marcus Müller wrote:
You leaking memory if you don't destroy plans is not surprising. It 
still makes no sense how you make an array, just to throw it away, as 
explained in my previous email. Your code really is not sensible, sorry!


Best regards,
Marcus

On 21.04.23 17:20, Grace Yeung wrote:
Thank you Marcus for taking the time to look through the backtrace 
error. ApplyFilterBank simply calls ForwardFT to perform the FFT and 
does not access the plan. It only uses the FFT output. I had to 
destroy the plan each time because I noticed memory was consistently 
growing when gnuradio was running the flowgraph. After adding the 
destroy_plan memory stops growing. Hope this makes sense. Thank you.


Grace

—​
Grace K. Yeung, MS
NorthWest Research Associates
301 Webster Street
Monterey, CA 93940
gr...@nwra.com
https://www.nwra.com
-- 


*From:* Marcus Müller 
*Sent:* Friday, April 21, 2023 7:10:43 AM
*To:* Grace Yeung 
*Subject:* Re: double free or corruption error when two instances of 
an OOT block using fftw is called in GRC

Hi Grace,

reading this backtrace, the error happens in

#29 0x7fffe99b04d5 in 
gr::customModule::probeExtract_impl::ApplyFilterBank


So, that's where you need to look, not sure whether the other code 
you shared helps.


But reading your other code: you seem to be making an FFT plan and 
destroying it *every
single time you do an FFT*. That's a terrible idea – planning an FFT 
typically takes much

longer than doing an FFT, even a really badly implemented one.
You also make a local array of plans, just to put that single plan 
in, then destroy it.
This indicates you've got a logical bug in your thinking about C++ 
scoping and lifetimes.
If I had to take a wild guess, you're trying to access a plan that 
does no longer exist in

your ApplyFilterBank.

Best regards,
Marcus

On 19.04.23 20:18, Grace Yeung wrote:
After some debugging, it appears grc is breaking inside of calls to 
fftw (see below). I am
simply trying to use fftw to perform a simple FFT transform 
(forward/backward). Is there
anything special I need to watch out for when using fftw with 
gnuradio? Any examples

available? Any advice would be much appreciated. Thanks!

Backtracing shows the crash happens in an fftw3f library function 
used in the code...


(gdb) bt
#0 __GI_raise (sig=sig@entry=6) at 
../sysdeps/unix/sysv/linux/raise.c:50
#1 >0x77dd1859 in 
__GI_abort () at

abort.c:79
#2 >0x77e3c26e in 
__libc_message
(action=action@entry=do_abort, fmt=fmt@entry=0x77f66298 "%s\n") 
at ../sysdeps/posix/libc_fatal.c:155
#3 >0x77e442fc in 
malloc_printerr
(str=str@entry=0x77f68628 "double free or corruption (fasttop)") 
at malloc.c:5347
#4 >0x77e45c65 in 
_int_free

(av=0x7fffac20, p=0x7fffac55a8d0, have_lock=0) at malloc.c:4266
#5 >0x7fffefe830d9 in 
?? () from

/lib/x86_64-linux-gnu/libfftw3f.so.3
#6 >0x7fffefe83c44 in 
?? () from

/lib/x86_64-linux-gnu/libfftw3f.so.3
#7 >0x7fffefe83fc4 in 
fftwf_mkplan_d ()

from /lib/x86_64-linux-gnu/libfftw3f.so.3
#8 >0x7fffefe88e5d in 
?? () from

/lib/x86_64-linux-gnu/libfftw3f.so.3
#9 >0x7fffefe8380b in 
?? () from

/lib/x86_64-linux-gnu/libfftw3f.so.3
#10 >0x7fffefe83a03 in 
?? () from

/lib/x86_64-linux-gnu/libfftw3f.so.3
#11 >0x7fffefe83fc4 in 
fftwf_mkplan_d ()

from /lib/x86_64-linux-gnu/libfftw3f.so.3
#12 >0x7fffefe8d95c in 
?? () from

/lib/x86_64-linux-gnu/libfftw3f.so.3
#13 >0x7fffefe8380b in 
?? () from

/lib/x86_64-linux-gnu/libfftw3f.so.3
#14 

Re: get_tags_in_range vs get_tags_in_window

[Marcus Müller]

Hello Jason,

no, there's no benefit. `get_tags_in_window` just calls `get_tags_in_range`; citing: 
gnuradio/gnuradio-runtime/lib/block.cc:


void block::get_tags_in_window(std::vector& v,
   unsigned int which_input,
   uint64_t start,
   uint64_t end)
{
d_detail->get_tags_in_range(v,
which_input,
nitems_read(which_input) + start,
nitems_read(which_input) + end,
unique_id());
}

So, sadly, no reduction of computational load (by the way, to evaluate computational load 
of the tagging system, we used to use `perf top -a` a lot).


Best regards,
Marcus

On 19.04.23 15:59, Jason Matusiak wrote:
I have a block that is making heavy use of tags: reading, reacting, and adding in new 
tags.  It seems like this block might be a little costly processor resource wise.  I am 
currently using get_tags_in_range, so I was wondering if there was a cost benefit to go 
with get_tags_in_window instead?


If it matters, this is a general_work block where I am taking in X samples, stuffing in 
random samples if there were any drops detected (to make up for them), and outputting Y 
samples.  The block isn't that complicated, but there is significant tag manipulation, and 
I think that that is costing me some.

Re: double free or corruption error when two instances of an OOT block using fftw is called in GRC

[Grace Yeung]

Got it. Thank you so much, Marcus!!

Grace

On 4/21/23 13:20, Marcus Müller wrote:

Hi Grace,

as said, this code shows that you don't understand how long (or 
rather: short) the object `plans` lives! This is a basic C++ issue: 
things created in local scope of a function end to exist at the end of 
the function. (I honestly think you'll need a bit of a better 
introduction to C++ – this will be easier for you with a solid 
understanding!)


So, your code needs fundamental rework:

- You currently always have to make a new FFTW plan, because your 
input might always be at a new address. That's bad. Instead, you must 
always put the samples into the same input buffer of your FFT. Then 
you can reuse the same plan over and over again.
- That means you have to rewrite ApplyFilterBank, so that it always 
puts the samples into the input buffer of your FFT.
- As soon as you've done that, you can use `gr::fft::fft` 
(which gives you an FFTw of type float that is forward=true), instead 
of doing **anything** with FFTW yourself.

- You can then just throw away your `ForwardFT` function. Instead,
  - add a `std::unordered_map> 
prepared_ffts` to

    your class's private members in probeExtract_impl.h, then
  - in `ApplyFilterBank`, you just do something like

    auto& [fft, already_there] = prepared_ffts.try_emplace(M, M, 
number_of_threads);

    auto& input_buffer = fft.get_inbuf();
    // TODO: put all input into input_buffer
    fft.execute()
    // output is now in fft.get_outbuf()

That's it. You don't have to care about any fftwf_init_threads, about 
any _delete_plan, you don't need to create the plans yourself.


Best,
Marcus

On 21.04.23 17:53, Grace Yeung wrote:


Hi Marcus,

Here is the ForwardFT() code below that is called by 
ApplyFilterBank() in the work function. So does it mean I should 
compute the plan only once in the constructor and use it throughout 
the work function by passing it to the ForwardFT() funciton? And 
declare _plans in the impl.h file?  Thanks.


Grace

   void probeExtract_impl::ForwardFT (vector &series, 
int M)

   {
   int exponent;
   fftwf_plan plans[MAX_FFT_EXP];

   exponent = (int) (log10((float) M)/log10(2.0) + 0.5);

   err_val = fftwf_init_threads ();
   if (err_val == 0)
   throw runtime_error("fftwf_init_threads failed");
   fftwf_plan_with_nthreads(NUM_FFTW_THREADS);

   pthread_mutex_lock (&fftw3_mutex);

   fftwf_make_planner_thread_safe();

   plans[exponent] = fftwf_plan_dft_1d (M,
reinterpret_cast (&series[0]),
reinterpret_cast (&series[0]),
    FFTW_FORWARD, 
FFTW_ESTIMATE);

   fftwf_execute (plans[exponent]);

   pthread_mutex_unlock (&fftw3_mutex);
   fftwf_destroy_plan (plans[exponent]);
   fftwf_cleanup_threads ();
   // Notes: destroying plans/cleaning up threads appear to 
eliminate growing memory issue 2/2023


   return;
   }


On 4/21/23 08:20, Grace Yeung wrote:
Thank you Marcus for taking the time to look through the backtrace 
error. ApplyFilterBank simply calls ForwardFT to perform the FFT and 
does not access the plan. It only uses the FFT output. I had to 
destroy the plan each time because I noticed memory was consistently 
growing when gnuradio was running the flowgraph. After adding the 
destroy_plan memory stops growing. Hope this makes sense. Thank you.


Grace

—​
Grace K. Yeung, MS
NorthWest Research Associates
301 Webster Street
Monterey, CA 93940
gr...@nwra.com
https://www.nwra.com
-- 


*From:* Marcus Müller 
*Sent:* Friday, April 21, 2023 7:10:43 AM
*To:* Grace Yeung 
*Subject:* Re: double free or corruption error when two instances of 
an OOT block using fftw is called in GRC

Hi Grace,

reading this backtrace, the error happens in

#29 0x7fffe99b04d5 in 
gr::customModule::probeExtract_impl::ApplyFilterBank


So, that's where you need to look, not sure whether the other code 
you shared helps.


But reading your other code: you seem to be making an FFT plan and 
destroying it *every
single time you do an FFT*. That's a terrible idea – planning an FFT 
typically takes much

longer than doing an FFT, even a really badly implemented one.
You also make a local array of plans, just to put that single plan 
in, then destroy it.
This indicates you've got a logical bug in your thinking about C++ 
scoping and lifetimes.
If I had to take a wild guess, you're trying to access a plan that 
does no longer exist in

your ApplyFilterBank.

Best regards,
Marcus

On 19.04.23 20:18, Grace Yeung wrote:
> After some debugging, it appears grc is breaking inside of calls 
to fftw (see below). I am
> simply trying to use fftw to perform a simple FFT transform 
(forward/backward). Is there
> anything special I need to watch out for when using fftw with 
gnuradio? Any examples

>