Packaging the json string as a vector of u8 integers worked like a charm, thank you for the suggestion!
Python code: https://gist.github.com/regulatre/38e7162d6f470ad0dbd77b171a69dc79 C++ block code: https://gist.github.com/regulatre/f264fb4751ca3bd8d6b98abf9792e95d Memory consumption by the python flowgraph is no longer showing signs of memory leakage (at least in the 4 hours it's been running :) On Fri, Mar 1, 2019 at 1:50 PM Müller, Marcus (CEL) <muel...@kit.edu> wrote: > Hi Brad, > > so, yes, your observation is correct: PMT symbols are/were meant to be > used as "identifiers", not as "data carriers"; the motivation behind > the hash table you find in pmt.cc is that there's only one instance of > any given PMT symbol, and thus, a simple address comparison suffices to > tell whether two PMT symbols are the same. > > You'll notice that on x86 (and presumably modern ARM) string comparison > is pretty fast, and that you'd need to do *a lot* comparisons to offset > the cost of hashing a symbol once. Anyway, yes, that table grows > unboundedly. > > Since your string isn't actually a "symbol", I'd recommend simply > encoding it safely (that's probably already the case), and putting it > in a uniform PMT vector of 8bit integers (u8vector). > > On a different note, there's actually "unintended" (as in: I don't > intend GNU Radio to have an unbounded hash map, but it's at least what > was originally intended) memory leaks with PMTs and tag_t on the > Python/C++ boundary, and there's quite some broken concepts within PMT. > > Long or medium term, we'll be replacing PMT with something that is > actually a common serialization format for usage in external software > (i.e. not for usage within the same flow graph), and ideally with the > unserialized universal container that comes with such a format. Stay > tuned. However, not happening in 3.8 or anything before. > > Best regards, > Marcus > > On Thu, 2019-02-28 at 14:48 -0500, Brad Hein wrote: > > > > In my gnuradio test application I’m attempting to pass a JSON formatted > string from my C++ custom block, to my python flow graph containing the > block. > > > > The String message are being received by my python flow graph, but > there’s a memory leak. Over time, my flowgraph RSS (memory footprint) grows > at about 32k/s up until the application runs out of memory. > > > > The leak didn’t start until after I retrofitted my flowgraph and custom > block with polymorphic type (PMT) based message passing. I’m passing a 200 > or so byte JSON string (as a symbol) from C++ block to python flow graph > about 60 times per second. > > > > Sleuthing through the pmt.cc code [1] I see the String (symbol) objects > are stored in a hash. I suspect what is happening is that since all of my > JSON strings are unique, they’re getting added to the hash and causing the > hash to grow unbounded over time. > > > > From what I can tell by reading the wiki [2], the approach I’m using is > the best way to get a string from my custom block and into my python > flowgraph. > > > > [1] > https://github.com/gnuradio/gnuradio/blob/master/gnuradio-runtime/lib/pmt/pmt.cc > > [2] https://www.gnuradio.org/doc/doxygen/page_pmt.html > > > > > > Sample c++ block code snippet from my work function: > > > > // Calculate metrics. Occasionally (60/second) we'll get back a JSON > formatted string of metrics. > > crossingData = calculateWaveCrossingMetrics(lastSampleValue,in[i]); > > > > if (crossingData.length() > 0) { > > pmt::pmt_t messageString; > > messageString = pmt::string_to_symbol(crossingData.c_str()); > > message_port_pub(polymorphicMessageDestination, messageString); > > > > > > Questions: > > > > 1. Is this the best way to get a string from my C++ block into my > python application? > > 2. Is there a way to pass my messages that doesn’t result in memory > growing unbounded? > > > > > > Thank you > > Brad > > > > _______________________________________________ > > Discuss-gnuradio mailing list > > Discuss-gnuradio@gnu.org > > https://lists.gnu.org/mailman/listinfo/discuss-gnuradio >
_______________________________________________ Discuss-gnuradio mailing list Discuss-gnuradio@gnu.org https://lists.gnu.org/mailman/listinfo/discuss-gnuradio
