Re: [Discuss-gnuradio] comments on stream tags and metadata storage
Hi Peter,
I'm on very limited time and internet, so let me just answer things
in-text, shortening where contributing to readability.
On 25.07.2014 23:37, Peter A. Bigot wrote:
> I've added five issues to cover the topics from my original email and
> followups.
>
> http://gnuradio.org/redmine/issues/698 proposes that the key of a
> stream tag be a namespaced identifier to avoid conflicts between
> individually-developed components.
That's a good idea, and actually, standarizing the way stream tags and
messages should be named and structured was one of the points of the
last dev call, and is thus work to be done.
>
> The other four are listed inline below with additional responses to
> Marcus:
>
> On 07/25/2014 08:46 AM, Marcus Müller wrote:
>> < issue at hand: inserting tags only seems save from within work, but
>> this is documented nowhere>
>>
>> sooo let me just whip up a few comments:
>> (1) That's a documentation issue, isn't it?
>
> I don't believe so.
>
> http://gnuradio.org/redmine/issues/699 expands on point (1), that
> add_item_tag() can only be safely called from within gr::block::work().
And I have to contradict Sean and you :)
Inserting tags *is* safe at any time. You get logical errors when you
don't take care that you don't attach tags to your stream that belong in
ranges that might already have been processed. This is logical; I do
think it would be a good idea to explicitely state it in the documentation.
However, there's no reason not to insert tags anywhere else, just make
sure that nitems_written() is bigger than your inserted tags' offsets.
>
>
>> Anyway, I'm not quite sure
>> you're right; the insertion of tags is mutexed IIRC, and the
>> get_tags_in_range() functionality, too, so once the user got his vector
>> of tags, that won't change anymore. There's the possibility that he
>> *misses* some of the tags for the range that get inserted after he
>> got_tags_in_range(), but that's only fair -- it's quite intuitive not to
>> insert tags after you've handed off the samples to which they would
>> belong to downstream tags.
>
> I wasn't concerned about tags across multiple calls to
> get_tags_in_range(), but for tags that are added to a stream after one
> call to work() and before the end of the next call to work().
I'm not sure I understand you correctly.
If (general_)work gets called, the samples that can be processed within
that call to work have left the upstream block's work function already.
If that upstream block decides that he wants to insert tags to samples
he produced in a prior call already, my comment to (1) and basic "how
the hell is this going to look causal to the block downstream" applies!
>
>
>> (3) I don't see the relation to the discussion, as you said ;) but that
>> sounds like a bug, so if you opened up a new thread or filed a bug at
>> gnuradio.org, that would be awesome :)
>
> http://gnuradio.org/redmine/issues/700 records point (3), that GRC
> parameter callbacks can be invoked multiple times as a result of a
> single user action due to the architecture of GUI and other components.
Thanks! Sounds hard to fix, though, if you say it's due to architecture.
>
>> (4) I'm fairly certain the buffers use a deque to store tags, not a map
>> of any kind. So maybe I'm misunderstanding you, or you misread code
>> somewhere?
>> I think what you're describing might be a bug in metadata_filesink, so
>> that might need some attention! see (3).
>
> Yes, the lead-in example was specific to the behavior of
> file_metadata_sink, but the fundamental requirement goes across the
> entire infrastructure. I'm hoping to see support for making that
> requirement stick.
It is *NOT* a requirement (so far). So far it is the rule that everyone
that expects multiple tags will do the sorting. And this makes sense --
computationally. I don't think we should "break" correctness of existing
blocks by taking away that requirement from the tag readers just to
shift it to the tag producers. So I strongly disagree with you here, sorry.
>
> http://gnuradio.org/redmine/issues/701 expands on points (2) and (4),
> that the infrastructure should promise to maintain all tags inserted
> by blocks in their original order (for any sample offset) and should
> document the situations where tags may be discarded by the
> infrastructure.
>
>
Also, I don't understand the bullet points at the end of your ticket #701.
* preserve order of insertion: GR does this, so how is this an issue?
* preserve incoming tag order: I'm not quite sure I get your point, but
if I do: GR does this, so how is this an issue?
* never discard a tag at the block that inserted it: Why would someone
discard a tag he inserted it? GR doesn't discard tags, so how is this an
issue?
* clearly document duplicate elimination process: duplicates are *not*
detected, so how is this an issue?
Seriously, tag insertion is *this* 2 lines of code
(gnuradio-runtime/lib/buffer_detail.cc):
void
buffer::add_item_tag(const tag_t &tag)
{
gr::thread
Re: [Discuss-gnuradio] comments on stream tags and metadata storage
I'm confused: are we to discuss these issues on the mailing list, or as comments on the wiki issues I created? I thought the latter was the right location. Putting detailed discussion in two places that do not link to each other is not a good approach. My comments below address interpretation and philosophy, not technical issues. On 07/26/2014 05:36 AM, Marcus Müller wrote: Hi Peter, I'm on very limited time and internet, so let me just answer things in-text, shortening where contributing to readability. On 25.07.2014 23:37, Peter A. Bigot wrote: I've added five issues to cover the topics from my original email and followups. http://gnuradio.org/redmine/issues/698 proposes that the key of a stream tag be a namespaced identifier to avoid conflicts between individually-developed components. That's a good idea, and actually, standarizing the way stream tags and messages should be named and structured was one of the points of the last dev call, and is thus work to be done. I've updated the issue with my proposed approach. If this is already being worked and people have an alternative architecture in mind, then please comment on that issue so I stop thinking about it. The other four are listed inline below with additional responses to Marcus: On 07/25/2014 08:46 AM, Marcus Müller wrote: < issue at hand: inserting tags only seems save from within work, but this is documented nowhere> sooo let me just whip up a few comments: (1) That's a documentation issue, isn't it? I don't believe so. http://gnuradio.org/redmine/issues/699 expands on point (1), that add_item_tag() can only be safely called from within gr::block::work(). And I have to contradict Sean and you :) Inserting tags *is* safe at any time. You get logical errors when you don't take care that you don't attach tags to your stream that belong in ranges that might already have been processed. This is logical; I do think it would be a good idea to explicitely state it in the documentation. However, there's no reason not to insert tags anywhere else, just make sure that nitems_written() is bigger than your inserted tags' offsets. I do not see how this allows me to guarantee association of a tag with a specific absolute sample. I can guess why you want to insert from outside work(), and could propose compromises, but I'd prefer to do that at http://gnuradio.org/redmine/issues/699 Anyway, I'm not quite sure you're right; the insertion of tags is mutexed IIRC, and the get_tags_in_range() functionality, too, so once the user got his vector of tags, that won't change anymore. There's the possibility that he *misses* some of the tags for the range that get inserted after he got_tags_in_range(), but that's only fair -- it's quite intuitive not to insert tags after you've handed off the samples to which they would belong to downstream tags. I wasn't concerned about tags across multiple calls to get_tags_in_range(), but for tags that are added to a stream after one call to work() and before the end of the next call to work(). I'm not sure I understand you correctly. If (general_)work gets called, the samples that can be processed within that call to work have left the upstream block's work function already. If that upstream block decides that he wants to insert tags to samples he produced in a prior call already, my comment to (1) and basic "how the hell is this going to look causal to the block downstream" applies! I don't think we disagree on this behavior, only on how to ensure that the original insertion doesn't get lost or misordered (and whether that's a problem for which the infrastructure should take some responsibility). (3) I don't see the relation to the discussion, as you said ;) but that sounds like a bug, so if you opened up a new thread or filed a bug at gnuradio.org, that would be awesome :) http://gnuradio.org/redmine/issues/700 records point (3), that GRC parameter callbacks can be invoked multiple times as a result of a single user action due to the architecture of GUI and other components. Thanks! Sounds hard to fix, though, if you say it's due to architecture. (4) I'm fairly certain the buffers use a deque to store tags, not a map of any kind. So maybe I'm misunderstanding you, or you misread code somewhere? I think what you're describing might be a bug in metadata_filesink, so that might need some attention! see (3). Yes, the lead-in example was specific to the behavior of file_metadata_sink, but the fundamental requirement goes across the entire infrastructure. I'm hoping to see support for making that requirement stick. It is *NOT* a requirement (so far). So far it is the rule that everyone that expects multiple tags will do the sorting. And this makes sense -- computationally. I don't think we should "break" correctness of existing blocks by taking away that requirement from the tag readers just to shift it to the tag producers. So I strongly disagree with you here, sorry. If t
Re: [Discuss-gnuradio] comments on stream tags and metadata storage
Hi Peter, On 26.07.2014 13:25, Peter A. Bigot wrote: > I'm confused: are we to discuss these issues on the mailing list, or as > comments on the wiki issues I created? I thought the latter was the > right location. Putting detailed discussion in two places that do not > link to each other is not a good approach. While I agree, I think the bug tracker might in the first place not have been the right choice for "architectural ideas", which fit better here. >> That's a good idea, and actually, standarizing the way stream tags and >> messages should be named and structured was one of the points of the >> last dev call, and is thus work to be done. > > I've updated the issue with my proposed approach. If this is already > being worked and people have an alternative architecture in mind, then > please comment on that issue so I stop thinking about it. Um, that's being worked on, but please don't take it personal when I say that I think the guys that have been using tags in the GNU Radio project should be the ones to agree on a way to standarize things, and do that at their own pace; I'm pretty sure your input will be valued, though! There have been things in discussion, and I guess there will be some communication regarding this soon, so I think it's safe to say it doesn't hurt if you don't spent too much time on this. > > I do not see how this allows me to guarantee association of a tag with a > specific absolute sample. That's always guaranteed - a Tag is a (offset, PMT) tuple. > I don't think we disagree on this behavior, only on how to ensure that > the original insertion doesn't get lost or misordered (and whether > that's a problem for which the infrastructure should take some > responsibility). I agree on disagreeing on whether this needs to be dealt with by the infrastructure enforcing this. Especially on the "introducing a complex restrictionary functionality just to stop people from doing logically wrong things". Basically, we're giving people vectors of void* to write samples to -- I think our users do pretty well at not writing to arbitrary memory locations, so we should trust ourselves not to produce tags for samples that happened in the past. If there is reason to do so, let them do so. I'm pretty confident that programmers that use tags are aware of the fact that the samples flow through their flow graph and that downstream blocks don't care about items that they already processed, and that they can't magically take into account tags you add while they are processing the samples that these tags are related to. > If there is information that is conveyed by receiving tags in a > non-monotonic offset order, I'd like to have more details. I don't think there is. But I can't rule it out, either. So it's a undecided, but applications might be relying on current behaviour. > > If the concern is that having the infrastructure maintain the tag order > I propose at http://gnuradio.org/redmine/issues/701 will introduce a > performance bottleneck, I'd like to see an argument. Preferably as a > comment on that issue. sorry to break your preference, but sorted inserting means that instead of constant time insertion you get time dependent on amount of tags, no arguing that, unless you find a better insertion algorithm for a sorted list ;) > > Otherwise, since I feel that a deterministic non-decreasing order of > arrival of tags simplifies implementing blocks that process those tags, agree on the simplifying aspect. > if you feel that every block that might take advantage of ordered > arrival should instead enforce it in their work() functions then we do > indeed disagree. So, indeed, we disagree. >> Also, I don't understand the bullet points at the end of your ticket >> #701. >> * preserve order of insertion: GR does this, so how is this an issue? > You missed the "within the same offset". Yes, I really do want > downstream blocks to be guaranteed that the tags they receive are sorted > by non-decreasing offset. No, I don't want to make every block that > inserts or receives tags have to do the sorting. GR does this, it's just appending to a deque. How would that mix up the order later on? If you do a std::stable_sort, the original order will also be kept. One more reason to let the user do this himself: He can decide whether stable_sorted, sorted or unsorted tags suffice. >> * never discard a tag at the block that inserted it: Why would someone >> discard a tag he inserted it? GR doesn't discard tags, so how is this an >> issue? > > The implementation of file_metadata_sink discards tags, as described. file_metadata_sink is not infrastructure, so I think we're mixing up subjects again. I was hoping you'd pack all metadata sink-related issues into one bug report and keep the discussion purely on the GNU Radio framework as is. > This suggests that GNU Radio as a holistic system does not have an > architectural requirement that metadata be preserved. I'd like to see > that resolution adopted. Me
Re: [Discuss-gnuradio] gr-fec LDPC comments / questions.
Hi Again,
> But in the paper I'm looking at (
> http://www.ldpc-codes.com/papers/encoding.pdf ) the T & E matrices are
> on the right side, which would match much better of course and
> honestly makes more sense to me since the reulting codeword often has
> the systematic bits first and then the parity bits.
Digging through the decoder I also found this assumption that the
codewords are provided with the parity bits first, then the systematic
bits :
for (index = 0; index < d_frame_size; index++) {
unsigned int i = index + n - d_frame_size; /* HERE */
int value = gsl_matrix_get(x, i, 0);
out[index] = value;
}
All the codes I know (granted it's only GMR and DVB) use the opposite
convention, so I think he'd be wise to either flip the convention in
the GR code, or make it configurable.
Do you know of any LDPC codes in common application where the parity
bits are placed/sent first ?
Cheers,
Sylvain
___
Discuss-gnuradio mailing list
Discuss-gnuradio@gnu.org
https://lists.gnu.org/mailman/listinfo/discuss-gnuradio
Re: [Discuss-gnuradio] comments on stream tags and metadata storage
Alright, guys, I think we're officially done with this thread! Seriously, while there's been good conversation here, there's too much going on to make any sane judgement on technical arguments. Let's move any discussion over to the issues tracker where Peter set them up and discuss there. Thanks! Tom On Sat, Jul 26, 2014 at 8:22 AM, Marcus Müller wrote: > Hi Peter, > > On 26.07.2014 13:25, Peter A. Bigot wrote: > > I'm confused: are we to discuss these issues on the mailing list, or as > > comments on the wiki issues I created? I thought the latter was the > > right location. Putting detailed discussion in two places that do not > > link to each other is not a good approach. > While I agree, I think the bug tracker might in the first place not have > been the right choice for "architectural ideas", which fit better here. > > > >> That's a good idea, and actually, standarizing the way stream tags and > >> messages should be named and structured was one of the points of the > >> last dev call, and is thus work to be done. > > > > I've updated the issue with my proposed approach. If this is already > > being worked and people have an alternative architecture in mind, then > > please comment on that issue so I stop thinking about it. > Um, that's being worked on, but please don't take it personal when I say > that I think the guys that have been using tags in the GNU Radio project > should be the ones to agree on a way to standarize things, and do that > at their own pace; I'm pretty sure your input will be valued, though! > There have been things in discussion, and I guess there will be some > communication regarding this soon, so I think it's safe to say it > doesn't hurt if you don't spent too much time on this. > > > > I do not see how this allows me to guarantee association of a tag with a > > specific absolute sample. > That's always guaranteed - a Tag is a (offset, PMT) tuple. > > > > I don't think we disagree on this behavior, only on how to ensure that > > the original insertion doesn't get lost or misordered (and whether > > that's a problem for which the infrastructure should take some > > responsibility). > I agree on disagreeing on whether this needs to be dealt with by the > infrastructure enforcing this. Especially on the "introducing a complex > restrictionary functionality just to stop people from doing logically > wrong things". > > Basically, we're giving people vectors of void* to write samples to -- I > think our users do pretty well at not writing to arbitrary memory > locations, so we should trust ourselves not to produce tags for samples > that happened in the past. If there is reason to do so, let them do so. > I'm pretty confident that programmers that use tags are aware of the > fact that the samples flow through their flow graph and that downstream > blocks don't care about items that they already processed, and that they > can't magically take into account tags you add while they are processing > the samples that these tags are related to. > > > If there is information that is conveyed by receiving tags in a > > non-monotonic offset order, I'd like to have more details. > I don't think there is. But I can't rule it out, either. So it's a > undecided, but applications might be relying on current behaviour. > > > > If the concern is that having the infrastructure maintain the tag order > > I propose at http://gnuradio.org/redmine/issues/701 will introduce a > > performance bottleneck, I'd like to see an argument. Preferably as a > > comment on that issue. > sorry to break your preference, but sorted inserting means that instead > of constant time insertion you get time dependent on amount of tags, no > arguing that, unless you find a better insertion algorithm for a sorted > list ;) > > > > Otherwise, since I feel that a deterministic non-decreasing order of > > arrival of tags simplifies implementing blocks that process those tags, > agree on the simplifying aspect. > > if you feel that every block that might take advantage of ordered > > arrival should instead enforce it in their work() functions then we do > > indeed disagree. > So, indeed, we disagree. > >> Also, I don't understand the bullet points at the end of your ticket > >> #701. > >> * preserve order of insertion: GR does this, so how is this an issue? > > You missed the "within the same offset". Yes, I really do want > > downstream blocks to be guaranteed that the tags they receive are sorted > > by non-decreasing offset. No, I don't want to make every block that > > inserts or receives tags have to do the sorting. > GR does this, it's just appending to a deque. How would that mix up the > order later on? If you do a std::stable_sort, the original order will > also be kept. One more reason to let the user do this himself: He can > decide whether stable_sorted, sorted or unsorted tags suffice. > > >> * never discard a tag at the block that inserted it: Why would someone > >> discard a tag he inserted it? GR do
Re: [Discuss-gnuradio] comments on stream tags and metadata storage
On 07/26/2014 08:59 AM, Tom Rondeau wrote: Alright, guys, I think we're officially done with this thread! Seriously, while there's been good conversation here, there's too much going on to make any sane judgement on technical arguments. Let's move any discussion over to the issues tracker where Peter set them up and discuss there. Thanks! Thank *you*. To close the loop: I've added comments to http://gnuradio.org/redmine/issues/699 and http://gnuradio.org/redmine/issues/701 that include links to Marcus' email with my response in excerpted context. Follow-ups there. Peter ___ Discuss-gnuradio mailing list Discuss-gnuradio@gnu.org https://lists.gnu.org/mailman/listinfo/discuss-gnuradio
Re: [Discuss-gnuradio] comments on stream tags and metadata storage
On 07/25/2014 08:22 PM, Peter A. Bigot wrote: > My work flow doesn't accommodate IRC very well but I've added #gnuradio > to my chat list (handle: pabigot) and will try to log in whenever I'm > working on topic. How does one find information on the next developers > call? I see some information on > http://gnuradio.org/redmine/projects/gnuradio/wiki/DevelopersCalls but > no schedule for future calls. Calls are every third Thursday of the month. If there's no agenda up, that usually means we haven't decided what we want to talk about. - M ___ Discuss-gnuradio mailing list Discuss-gnuradio@gnu.org https://lists.gnu.org/mailman/listinfo/discuss-gnuradio
[Discuss-gnuradio] Problem with large number of inputs.
Hi all, I've used to have a block with large number of inputs, say 24, but I think GNURadio has a problem with memory allocation because when I use the block it's suddenly jump out of the program with "The program has unexpectedly finished." (the above message isn't for GNURadio) However, when I decrease the number of inputs to 8, it's fine and works properly! This is so strange to me. Any idea? Best, Mostafa ___ Discuss-gnuradio mailing list Discuss-gnuradio@gnu.org https://lists.gnu.org/mailman/listinfo/discuss-gnuradio
Re: [Discuss-gnuradio] Problem with large number of inputs.
Hi Mostafa, indeed, this sounds strange. However, with but the information you offered, it will be hard to determine what goes wrong. Would you have more information, e.g. when exactly this happens, is this C++ or python, do you have source code, in which code line the error occurs (hint: [1])? Greetings, Marcus [1] http://gnuradio.org/redmine/projects/gnuradio/wiki/TutorialsDebugging, follow the gdb instructions On 26.07.2014 19:49, Mostafa Alizadeh wrote: > Hi all, > > I've used to have a block with large number of inputs, say 24, but I think > GNURadio has a problem with memory allocation because when I use the block > it's suddenly jump out of the program with > > "The program has unexpectedly finished." > > (the above message isn't for GNURadio) > > > However, when I decrease the number of inputs to 8, it's fine and works > properly! > > This is so strange to me. > > > Any idea? > > > Best, > > Mostafa > > > > ___ > Discuss-gnuradio mailing list > Discuss-gnuradio@gnu.org > https://lists.gnu.org/mailman/listinfo/discuss-gnuradio ___ Discuss-gnuradio mailing list Discuss-gnuradio@gnu.org https://lists.gnu.org/mailman/listinfo/discuss-gnuradio
Re: [Discuss-gnuradio] Problem with large number of inputs.
Hi Marcus,
You're right. I didn't clarify the problem.
Actually in C++, I wrote:
block_impl::block_impl(bool normal, int group)
: gr::block("phich_grouping",
gr::io_signature::make(1, group * N_FS(normal), sizeof
(gr_complex)),
gr::io_signature::make(1, 1, sizeof(gr_complex))),
// N_FS function returns 8 or 4 depends on its input.
// in the work function :
// d_normal and d_group are defined in the .h file
const gr_complex *in[d_N_phich_group * N_FS_phich(d_normal_cp)];
for (int i=0; i___
Discuss-gnuradio mailing list
Discuss-gnuradio@gnu.org
https://lists.gnu.org/mailman/listinfo/discuss-gnuradio
