On 07/25/2014 09:36 AM, Tom Rondeau wrote:
On Fri, Jul 25, 2014 at 10:02 AM, Martin Braun <martin.br...@ettus.com
<mailto:martin.br...@ettus.com>> wrote:
Hey Peter,
you did come to the right place! And no-one here claims that
stream tags
are "finished". On the other hand, you should realize that you've
brought up something pretty specific, and the few people who are
actually qualified to answer all of your questions might be pretty
busy.
I would like to invite you to our monthly developer's calls, where
things like these can be addressed in a more interactive way. Also, if
you believe you've found a bug, submit it to our tracker, so we can
track it. If you post your intent to fix it, we'll be very happy, and
certainly won't stop you from fixing something! This would also
split up
your lengthy email into sizable chunks.
Now, I realize you did not only post what you thought were bugs, but
also suggestions on concepts. Again, a good place to ask those are the
dev calls, and depending on time, IRC is actually a decent place to
discuss this.
That said, a few short comments (probably not what you wanted):
- Tag ordering: We simply don't have a rule that tags need to be
ordered. Blocks that require ordered tags run a quick std::sort()
usually, but most blocks don't care.
- Since we use absolute offsets, that's usually not a problem
- If file_meta_sink doesn't sort(), that might actually be a bug, and
we'd appreciate an issue (and a fix :)
- If a tag position is 'inaccurate', it is still unambiguously
mapped to
an item, and it's up to the dev to handle an 'inaccuracy'
-- M
Peter,
Basically everything that Martin said. I haven't even read your email
in its entirety, yet. It was just too much at one time to start any
decent conversation about any of the topics. It's much easier to
address issues one at a time in an email thread. And yes, use our
bug/feature tracker.
And as for tags, we've had lots of people contribute complaints, but
no one's contributed solutions or patches :)
Tom
Thanks for the responses.
I perceive there's some variation in expectations not only for how tags
should behave, but how they do behave as currently implemented. I'll
reproduce whatever I see as problematic behavior and create bug reports
so there's context for targeted discussion among a smaller audience.
Much of the rest of my essay related to capabilities/behavior GNU Radio
doesn't support them now, but that need to be considered at the
architecture level if they're ever to be supported. Those points might
be harder to work through, but I'll do that via issues as well.
I am coming in new here, but my tentative conclusion is that it'll be
difficult to achieve my goals with the existing tag infrastructure. I
found few obvious uses of it in the baseline repository. Part of my
goal in this discussion was to get a feel for how much it's being
actively used for real applications (i.e, has the ship sailed/horse left
the barn/pick your metaphor). I still don't have a sense for that.
My work flow doesn't accommodate IRC very well but I've added #gnuradio
to my chat list (handle: pabigot) and will try to log in whenever I'm
working on topic. How does one find information on the next developers
call? I see some information on
http://gnuradio.org/redmine/projects/gnuradio/wiki/DevelopersCalls but
no schedule for future calls.
Peter
On 07/25/2014 01:00 PM, Peter A. Bigot wrote:
> I'd hoped my comments below would start a more extensive dialog
on GNU
> Radio's metadata infrastructure. Several years experience that I
have
> with this capability in a non-commercial C++ DSP framework
suggests many
> enhancements in flow, representation, and utilities.
>
> I have a slight itch to contribute to a solution, but without
community
> involvement can't hope to provide anything mergable. Is this
simply not
> something anybody feels needs to be addressed, or did I ask in
the wrong
> forum?
>
> Peter
>
> On 07/17/2014 05:11 PM, Peter A. Bigot wrote:
>> Some comments after playing with stream tags and metadata this
>> afternoon.
>>
>> (1) Although the discussion of stream tag insertion hints that this
>> should be done within the scheduler's call to work() it could
be more
>> clear that doing it in any other context can result in race
conditions.
>> (I did think I saw it stated more clearly somewhere, but can't find
>> that now, so maybe this point has been addressed.)
>>
>> (2) In the current implementation it's further necessary that
tags be
>> added to an output in monotonic non-decreasing offset order.
>> file_meta_sink does not sort the return value from
get_tags_in_range(),
>> and emits all data up to the timestamp of the next tag, so a
subsequent
>> tag with an earlier offset is dropped from the archive.
>>
>> (I note that tagged_file_sink() does sort the tags it receives
in one
>> case, but not in others.)
>>
>> I don't see this requirement on ordered generation documented.
In some
>> cases, it may be inconvenient to do this, e.g. when a block's
analysis
>> discovers after-the-fact that something interesting can be
associated
>> with a past sample. Similarly, a user might want a block to
associate
>> a tag with sample that not yet arrived, to notify a downstream
block
>> that will need to process the event.
>>
>> A simple solution for the infrastructure is to require that
tags only be
>> generated from within work(), with offsets corresponding to samples
>> generated in that call to work(), and in non-decreasing offset
order
>> (though this last requirement could be handled by
add_item_tag()). The
>> developer must then handle the too-late/too-early tag associations
>> through some other mechanism, such as carrying the effective
offset as
>> part of the tag value.
>>
>> (3) Qt GUI Range with widget Counter + Slider invokes callbacks
twice,
>> even if the value itself was set exactly once through the
counter text
>> entry. If the callback records the change by queuing a stream
tag for
>> addition to the output, multiple tags with the same
offset/key/value
>> will be generated.
>>
>> There are ugly solutions to this but it's probably sufficient
to note
>> somewhere that it can happen. It's really not specific to
tags, but is
>> clearly visible in that case.
>>
>> (4) The in-memory stream of tags can produce multiple settings
of the
>> same key at the same offset. However, when stored to a file
only the
>> last setting of the key is recorded.
>>
>> I believe this last behavior is incorrect and that it's a
mistake to use
>> a map instead of a multimap or simple list for the metadata
record of
>> stream tags associated with a sample.
>>
>> One argument is that it's critical that a stream archive of a
processing
>> session faithfully record the contents of the stream so that
re-running
>> the application using playback reproduces that stream and thus the
>> original behavior (absent non-determinism due to asynchrony). This
>> faithful reproduction is what would allow a maintainer to
diagnose an
>> operational failure caused by a block with a runtime failure
when the
>> same tag is processed twice at the same offset. This is true
even if
>> the same key is set to the same value at the same sample offset
multiple
>> times, which some might otherwise want to argue is redundant.
>>
>> A corollary argument is that the sample number at which an
event like a
>> tuner configuration change occurs usually can't be exactly
associated
>> with a sample; the best estimate is likely to be the index of
the first
>> sample generated by the next call to work. But depending on
processing
>> speed an application might change an attribute of a data source
multiple
>> times before work was invoked. The effect of those
intermediate changes
>> may be visible in the signal, and to lose the fact they occurred by
>> discarding all but the last change affects both reproducibility and
>> interpretation of the signal itself.
>>
>> (5) All stream tags are placed in the extras block, and when a
segment
>> is completed file_meta_sink will generate a new header. The
new header
>> contains copies of the unique tags, but updates their offsets
to be the
>> start of the new segment.
>>
>> This is incorrect as the original stream did not have those tags
>> associated with those samples, so re-playing will introduce a
behavioral
>> difference. For example, a tag that is meant to be associated
with the
>> start of a packet will be duplicated at an offset that is
probably not
>> the start of a packet.
>>
>> Solutions include (a) leave the original offset setting for
tags in the
>> extras section when they're reproduced in a new segment, even
though
>> that offset is not present in the segment; (b) treat stream tags as
>> ephemeral and do not persist them in the extras section when
generating
>> a new segment; (c) extend the add_item_tag API to record
whether the
>> tag is ephemeral or persistent. Offhand I can see no argument
>> supporting persisting a tag and updating its offset, and only
rare cases
>> where it's appropriate to replicate outdated information in a new
>> segment, so (b) seems to be the right move.
>>
>> All the above is based on my understanding and expectations of how
>> stream tags are/should be used. If my understanding is mistaken,
>> please let me know.
>>
>> Peter
>>
>>
>> _______________________________________________
>> Discuss-gnuradio mailing list
>> Discuss-gnuradio@gnu.org <mailto:Discuss-gnuradio@gnu.org>
>> https://lists.gnu.org/mailman/listinfo/discuss-gnuradio
>
>
> _______________________________________________
> Discuss-gnuradio mailing list
> Discuss-gnuradio@gnu.org <mailto:Discuss-gnuradio@gnu.org>
> https://lists.gnu.org/mailman/listinfo/discuss-gnuradio
_______________________________________________
Discuss-gnuradio mailing list
Discuss-gnuradio@gnu.org <mailto:Discuss-gnuradio@gnu.org>
https://lists.gnu.org/mailman/listinfo/discuss-gnuradio
_______________________________________________
Discuss-gnuradio mailing list
Discuss-gnuradio@gnu.org
https://lists.gnu.org/mailman/listinfo/discuss-gnuradio
_______________________________________________
Discuss-gnuradio mailing list
Discuss-gnuradio@gnu.org
https://lists.gnu.org/mailman/listinfo/discuss-gnuradio
