Hi Gonzalo,
On 29.02.2016 04:58, Gonzalo Arcos wrote:
> I have seen that most of the buffers are almost full on average
> (80-89%), however, that does not help me to know whether a block was
> blocked from pushing data into the buffer because downstream blocks
> did not read fast enough.
Hm, while uniform usage is usually a good sign, that much average fill
isn't good. You say "most" blocks; what's downstream of these high fill
buffers?
>
> From your original mail:
>
> "File Source is only allowed to produce so many items that the write
> pointer doesn't advance beyond the minimum read pointer, because in
> that case, it would overwrite samples that a downstream block hasn't
> consumed."
>
> The maximum distance between the write pointer and the minimum read
> pointer, is percentage based? Is a fixed item value? This is because i
> tried to play around with the buffer sizes but did not notice any
> change in the performance of the flowgraph, either for good or bad.
I don't really understand your question. The point is that the position
of the write pointer might never pass the read pointer(s). The
difference between the "most behind" read pointer and the write pointer
is used to calculate noutput_items for the block's (general_)work call.
In fact, your block is asked "to generate this much output, how much
input would you need?" via its "forecast" method with exactly that
difference. If your forecast says it needs more input than available,
the number is halved, and "forecast" is called again, until forecast
only demands as much input as available. At that point, general_work is
called with noutput_items set to that number.
>
> You said that a buffer with size 0, will use the default gnu radio
> size. I would like to ask, is this a fixed value taken from a config
> file?
umm... Buffer generation. Wait... Here it is: [1] and used here [2].
So, it defaults to 64KB buffers. That aligns nicely witht the
observation that complex number buffers are typically 8192 items long
(complex = float I + float Q = 32bit + 32bit = 8B; 8192 * 8 = 64K).
If you have other restrictions (e.g. set_min_output_buffer, or item size
not factor of page size (4KB)), you might get different sizes, though.
> or gnuradio automatically asses the system memory and sets a default
> buffer appriately? For example, if i would like to give gnuradio 6 GBs
> of RAM for the buffers, then upstream blocks should not block because
> downstream blocks are slow processing the buffer data, at least for a
> big amount of items.
Hm, that would be a nice feature, at least for experimentation
(enlarging all the buffers might have pretty ugly effects on latency, so
I wouldn't recommend doing it unless you know your sample rate *
buffer_size/2 * blocks in critical path are significantly less than your
tolerable latency). I don't think GNU Radio has that feature, though :)
I'd say, in general, it's a good idea to manually enlarge the output
buffers of the blocks upstream of CPU-intense blocks, but to keep
granularity small in the rest of the flow graph.
Feel free to experiment with [1]; you could remove the assignment of the
#define constant to s_fixed_buffer_size and replace it with a call to
const unsigned int s_fixed_buffer_size =
prefs::singleton()->get_long("DEFAULT", "buffer_size",
GR_FIXED_BUFFER_SIZE);
and add a "buffer_size = X" to your .gnuradio/config.conf under the
[default] clause¹.
Best regards,
Marcus
¹ um, yeah, that should make the staticness a bit conceptually
questionable, which should lead to us renaming the variable, but... hey,
experimentation!
[1]
https://github.com/gnuradio/gnuradio/blob/master/gnuradio-runtime/lib/flat_flowgraph.cc#L41
[2]
https://github.com/gnuradio/gnuradio/blob/master/gnuradio-runtime/lib/flat_flowgraph.cc#L134
> 2016-02-26 14:48 GMT-03:00 Gonzalo Arcos <gonzaloarco...@gmail.com
> <mailto:gonzaloarco...@gmail.com>>:
>
> I will investigate and try this. Thank you so much Marcus!
>
> 2016-02-26 12:09 GMT-03:00 Marcus Müller <marcus.muel...@ettus.com
> <mailto:marcus.muel...@ettus.com>>:
>
> Hi Gonzalo,
>> However i noticed that after the optimization, the sum of all
>> blocks percentage of processing time is not 100%, this is
>> when i started evaluating the possibility that gnu radio
>> scheduler or gnuradio framework or the relationship between
>> blocks is what is preventing me to achieved my desired rate,
>> and it is not each block processing time anymore.
> Well, in GNU Radio, every block runs in its own thread, so you
> can basically can get up to the number of CPU cores * 100% as
> total consumption.
>> However, i do not have an easy way to determine the "amount
>> of time a block has been blocked because another block in the
>> downstream has not read the input buffer fast enough for the
>> first block to be able to write on the buffer".
> The performance monitor should be able to tell you how full
> your buffers are on average; watch out for "full" buffers:
> they're upstream from your bottlenecks.
> Also, look for the average rate (as reported e.g. by the
> probe_rate), and compare that value with the (average work
> items/average work call time). That kind of gives you a "block
> duty cycle".
>
> Best regards,
> Marcus
>
>
>
> On 02/26/2016 01:51 PM, Gonzalo Arcos wrote:
>>
>> Hi Marcus,
>>
>> Thanks a lot for your answer! So now having confirmed this,
>> comes the real question :)
>>
>> I am trying to improve the performance of a flowgraph. My
>> performance "metric" is the rate at which a file in my
>> filestystem grows, since my flowgraph starts with a file
>> source and ends with a file sink. This means that after doing
>> some processing, the flowgraph outputs the result to a file,
>> so if the file grows faster, then the flowgraph is executing
>> faster and i achieve a faster transfer rate, even if i then
>> change the file sink for another sink (i.e. usrp).
>> The transfer rate is very important to me, because im
>> evaluating the performance of the flowgraph compared to the
>> implementation that already exists in hardware, which i know
>> exactly the capable transfer rate.
>>
>> So after using the performance monitor and detecting some
>> blocks that do a lot of processing, ive optimized those blocks.
>>
>> However i noticed that after the optimization, the sum of all
>> blocks percentage of processing time is not 100%, this is
>> when i started evaluating the possibility that gnu radio
>> scheduler or gnuradio framework or the relationship between
>> blocks is what is preventing me to achieved my desired rate,
>> and it is not each block processing time anymore.
>>
>> However, i do not have an easy way to determine the "amount
>> of time a block has been blocked because another block in the
>> downstream has not read the input buffer fast enough for the
>> first block to be able to write on the buffer".
>>
>> It is also not as simple as connecting a file sink in an
>> intermediate point in the flowgraph delete the rest of the
>> blocks, and measure the transfer rate against this new file
>> sink. This is because in my flowgraph interpolation and
>> decimation occurs at different blocks, resulting in
>> making the file size an unrealiable metric in these cases.
>>
>> In the only case this is valid, is at the very end of the
>> flowgraph, when the data has been restored to its original
>> content, and therefore, any output by from the final block
>> corresponds to an input byte.
>>
>> I hope i expressed myself correctly. Do you have any tip on
>> how to figure out why my flowgraph at a determine rate?
>> Should the time a flowgraph executes be always the same to
>> the sum of the time all blocks consume?
>>
>>
>> ---------- Forwarded message ----------
>> From: *Marcus Müller* <marcus.muel...@ettus.com
>> <mailto:marcus.muel...@ettus.com>>
>> Date: 2016-02-26 6:52 GMT-03:00
>> Subject: Re: [Discuss-gnuradio] How to specify maximum
>> size of input buffers on blocks
>> To: discuss-gnuradio@gnu.org
>> <mailto:discuss-gnuradio@gnu.org>
>>
>>
>> Hi Gonzalo,
>>
>> these are the mails I like most :)
>> On 25.02.2016 01:11, Gonzalo Arcos wrote:
>>> Suppose i have a file source, connected to 2 blocks,
>>> one, lets call it Block A, does always return instantly
>>> after general work is called, in other words, it will
>>> never consume an input item. The other one, say its
>>> block B, is the identity block, it consumes and output
>>> exactly what it receives as input.
>>
>>>
>>> The output of the first block is a null sink (since i
>>> know it will not produce any output), and the output of
>>> the second block is a file sink.
>> So:
>>
>> /->A->Null Sink
>> File Source -|
>> \->B->File Sink
>>
>>>
>>> What i am experiencing at running this flowgraph, is
>>> that block B will work for the very first seconds (or
>>> centiseconds), and will eventually block. This is
>>> because A has never consumed an input item, so i guess
>>> A's input buffer is full. Because of this, the file sink
>>> cannot push any more items further into the flowgraph,
>>> resulting in B not having any new input items to process.
>> Exactly!
>> So the mechanism below is: the output buffer of File
>> Source is the input buffer of A and the input buffer of
>> B. No memory duplication here.
>> File Source has a buffer writer with a write pointer, and
>> A and B have their own read pointers pointing into that
>> buffer.
>> When File Source produces N items, the write pointer
>> advances by N. Similarly, when A consumes M items, A's
>> read pointer advances by M.
>> When calling (general_)work, the input_items buffer(s) is
>> (are) really just a pointer (start_of_buffer + read
>> pointer). Equivalently, the output_items buffer(s) is
>> (are) really just pointing to the write pointer.
>>
>> File Source is only allowed to produce so many items that
>> the write pointer doesn't advance beyond the minimum read
>> pointer, because in that case, it would overwrite samples
>> that a downstream block hasn't consumed.
>>
>>> So, my question is, how big is the input buffer of A?
>> Depends. Typically (64bit Linux), 8192 complex items get
>> allocated, but that really depends on various factors.
>>> Is that customizable?
>> Yes! For example, as you noticed, in the GRC, open the
>> "advanced" tab in a block property; there's a "Min Output
>> Buffer" and a "Max Output buffer field". There's
>> corresponding methods to set these sizes in the gr::block
>> class[1].
>>> Can i increase the size, so the flowgraph wont block at
>>> the first seconds, but at a minute, for example?
>> Yes; but that might, depending on the rate in which the
>> file source produces samples, be *a whole lot* of memory!
>> I recommend not doing that, but instead:
>>
>> 1. Make A consume the items it has read (because, as far
>> as I can tell, you don't actually need them,
>> afterwards, do you?)
>> 2. implement your "end the running of this flowgraph
>> after X samples" by adding a "head" block, which has
>> the functionality to simply consume all N input
>> samples, copy them from its input to its output
>> buffer, and then consume(N). If sum(N) == X, it says
>> it's done and thus leads to the flowgraph be shut down
>>
>>> Can i specify a policy for a block that if it reaches X
>>> amount of samples in its input buffer to drop some of
>>> the input?
>> You could just write a block that does that for you.
>>>
>>> Ive seen that each block has minimum output buffer and
>>> maximum output buffer in grc. However, i do not see any
>>> option regarding its INPUT buffer.
>> Because there's no such thing as a dedicated input buffer
>> :) because every input buffer is in fact the output
>> buffer of the upstream block.
>>>
>>> Another thing i thought is that A's input buffer is the
>>> file source output buffer, therefore by adjusting the
>>> output buffer of the file source, i am adjusting A's
>>> (and B's) input buffer. However, the output buffer of
>>> the file source is 0, so i guess its "infinite".
>> No, 0 just instructs GRC to not write a line containing a
>> "set_min_output_buffer" call[2], so GNU Radio uses the
>> default.
>>
>> Best regards,
>> Marcus
>>
>> [1]
>>
>> https://gnuradio.org/doc/doxygen/classgr_1_1block.html#a9d10e3f6747f91b215abe81b60a003d5
>> [2]
>>
>> https://github.com/gnuradio/gnuradio/blob/master/grc/python/flow_graph.tmpl#L230
>>>
>>>
>>> _______________________________________________
>>> Discuss-gnuradio mailing list
>>> Discuss-gnuradio@gnu.org <mailto:Discuss-gnuradio@gnu.org>
>>> https://lists.gnu.org/mailman/listinfo/discuss-gnuradio
>>
>>
>> _______________________________________________
>> Discuss-gnuradio mailing list
>> Discuss-gnuradio@gnu.org <mailto:Discuss-gnuradio@gnu.org>
>> https://lists.gnu.org/mailman/listinfo/discuss-gnuradio
>>
>>
>>
>
>
>
_______________________________________________
Discuss-gnuradio mailing list
Discuss-gnuradio@gnu.org
https://lists.gnu.org/mailman/listinfo/discuss-gnuradio
