If you decide to go that route I'm happy to discuss possible re-use of
libsigmf and/or lessons learned from the trenches offline (I happen to be
using it a *lot*). It's not *really* specific to SigMF. It's really just
about getting c++ struct types easily converted to/from python. External
dependencies are flatbuffers and Modern Json (nlohmann/json on github). It
doesn't care much about the binary serialization but does make working with
json-namespaced objects and moving between the c++ struct types and json
real nice. In a truly ideal world we actually wouldn't need libsigmf
because flatbuffers would include this parsing + namespace handling out of
the box and that may be possible if flatbuffers devs woulf find this
functionality valuable.
Nathan
On Sun, Sep 29, 2019 at 10:04 AM Marcus Müller <mmuel...@gnuradio.org>
wrote:
> Wow, thanks. Yeah, that clearly reinforces flatbuffers as tool of
> choice.
>
> Tbh, I'd just assume that in all these formats, being tight-packing by
> default, std::complex<float> can just be represented by the equivalent
> of struct {float re; float im;} complex;.
>
>
> On Sun, 2019-09-29 at 09:32 -0500, West, Nathan wrote:
> > While developing libsigmf (https://github.com/deepsig/libsigmf) I
> > went through a similar experience. The objective was to have painless
> > json file/string <-> real c++ type (this means the serialization
> > thing had to generate a slimmed down type that is effectively a
> > struct of vector, int, float, etc). I looked at cap'n proto,
> > flatbuffers, protobuf, and some others that weren't as
> > mature/supported. I actually started with cap'n proto and had to
> > switch to flatbuffers.
> >
> > Other bonuses I encountered were the whole "zero-time serialization"
> > (In my experience people love making fun of cap'n proto for this, but
> > it's true for all of the popular architectures and then you pay for
> > shuffling bytes on more exotic architectures) and the sort of random-
> > access parsing. A big perk of cap'n proto was it actually came with
> > json-parsing to flatbuffer type *and* had the ability to generate a
> > sane struct definition.
> >
> > I eventually switched to flatbuffers. Here's a randomly-ordered list
> > of why
> >
> > * cap'n proto's author is no doubt an incredible programmer. However,
> > he made some really weird choices. Like wrote his own string class in
> > libkj and made all of cap'n proto depend on it. And it's not all that
> > friendly with c++ string. So you wind up having to convert every
> > single string in to a kj::string just for serialization (and vice
> > versa)
> > * libkj also implements its own memory allocators, filesystem library
> > (ala boost filesystem / c++ filesystem), mutexes, and threads. I'm
> > not capable of vetting those implementation but beg to be asked a big
> > old "WHY?!"
> > * cap'n proto is incredibly strict (in CS parlance I guess you can
> > call it opinionated) on protocol description language. It enforces
> > things such as FirstletterMustBeCapital
> > * I also felt the syntax of moving around these objects was a bit
> > awkward
> >
> > I think there may be some other less appalling things that I've
> > struck from my memory as "writing your own string that's incompatible
> > with the rest of c++" is a big enough grievance that I'm only just
> > now starting to forgive gcc dual abi for (but at least even that is
> > at the ABI level which is always hard to get right). Flatbuffers
> > offered pretty much the same promises as cap'n proto but without the
> > "reinvent the standard library first" mindset and let's you make your
> > own choices about how to name your fields, etc. I will say cap'n
> > proto had a nice-looking extension capability to their parsing which
> > I haven't found a good alternative for.
> >
> > The downside is flatbuffers is released by google/android's
> > benevolence and somewhat competes with protobufs which is clearly
> > their darling. I can't tell how much anyone else cares about
> > flatbuffers although the *major* advantage to flatbuffers that I
> > couldn't find in protobuf (which made me choose flatbuffers) is the
> > struct code generator so you get out pretty normal struct's from
> > their code generator that look like something you would write if you
> > did it by hand.
> >
> > The downside is I don't know what complex<T> support looks like in
> > any of these things.
> >
> > Cheers,
> > Nathan
> >
> > On Sun, Sep 29, 2019 at 5:21 AM Marcus Müller <mmuel...@gnuradio.org>
> > wrote:
> > > Hi Nick,
> > >
> > >
> > > On Sat, 2019-09-28 at 15:54 -0700, Nick Foster wrote:
> > > > I think it's wise to avoid hitching our metaphorical horse to any
> > > > software project we can't absolutely depend on; questionable
> > > > dependency choices have bit us, as you mention, many times.
> > >
> > > Sad but true.
> > >
> > > > With that (somewhat) in mind, what about Flatbuffers as an
> > > > alternative to Cap'n Proto? Seems much more actively maintained,
> > > more
> > > > widely used, and designed with the same goals in mind. No parsing
> > > > step required, so performance is good. Memory use seems to be
> > > > similarly efficient.
> > >
> > > See, that's the kind of input I was looking for!
> > > So, what I don't like about flatbuffers was the lack of field IDs
> > > in
> > > the schema; relying on new fields only being added to the end of a
> > > structure.
> > >
> > > However, thanks to your encouragement I've revisited that and:
> > > Flatbuffers *does* do IDs, if you want (it's slightly awkward, but
> > > all
> > > formal structure definition languages are, aside from ASN.1 ;) ).
> > >
> > > Regarding efficiency: https://github.com/thekvs/cpp-serializers
> > > seems
> > > to agree with your assessment.
> > >
> > > What flatbuffers really got going for it compared to the Cap'n is
> > > that
> > > docs are structured instead of "cool feature-oriented". What Cap'n
> > > Proto has is the more sympathetic code base, but maybe I shouldn't
> > > be
> > > applying too many human qualities to code bases. gnuradio-
> > > runtime/lib/
> > > gets jealous everytime I do that.
> > >
> > > Best regards,
> > > Marcus
> > >
> > > >
> > > > Now, I haven't actually used Flatbuffers before, so anyone who
> > > has
> > > > experience with them (or your other favorite serialization
> > > library!),
> > > > please chime in here.
> > > >
> > > > Nick
> > > >
> > > > On Sat, Sep 28, 2019 at 2:02 PM Marcus Müller <
> > > mmuel...@gnuradio.org>
> > > > wrote:
> > > > > To avoid the setting in of post-Conference downs:
> > > > >
> > > > > Dearest SDR Community with the Highest count of "Awesome" in
> > > the
> > > > > Universe,
> > > > >
> > > > > discussions at GRCon have made it abundantly clear that PMT
> > > needs
> > > > > to be
> > > > > replaced¹.
> > > > > Main issues with PMT are:
> > > > >
> > > > > * broken type system,
> > > > > * low suitability as tool for internal packeted data,
> > > > > * lack of interoperability with anything but the languages we
> > > > > support
> > > > > natively,
> > > > > * platform-dependency of the serialization format,
> > > > > * bugs.
> > > > >
> > > > > (to a far lesser degree: performance.)
> > > > >
> > > > > Having talked to multiple domain experts in networked systems
> > > and
> > > > > heterogenous cooperative platforms, I've come to the
> > > conclusion,
> > > > > furthermore, that in the interest of RPC'ing everything², the
> > > > > structure
> > > > > of messages exchanged between blocks should be predefined.
> > > > >
> > > > > In this sense, PMT's (limited) ability do describe itself is of
> > > no
> > > > > help
> > > > > – instead, it introduces ambiguity.
> > > > >
> > > > > The dominant choice for binary serialization with known
> > > schemata is
> > > > > Proto Buffers; however, they require deserialization of a whole
> > > > > message
> > > > > if there's substructures before the field you want to access³.
> > > > >
> > > > > Cap'n Proto[1] (by one of the original authors of protobufs)
> > > seems
> > > > > to
> > > > > be a suitable alternative, and comparing it to its competitors,
> > > it
> > > > > seems to have achieved a significant level of maturity and has
> > > at
> > > > > least
> > > > > one large-scale adopter⁴.
> > > > >
> > > > > Among its advantages seems to be that operations on the message
> > > > > happen
> > > > > in-place (i.e. there's actually a compatible C++ struct
> > > > > representation
> > > > > of the memory layout), which rendered it my favourite library
> > > for
> > > > > this
> > > > > job.
> > > > >
> > > > > Now, problem: I've never personally worked with Cap'n Proto, or
> > > > > protobuf, or any modern, non-self-describing serialization
> > > library;
> > > > > I've got minimal experience with MessagePack, but as said,
> > > that's
> > > > > not
> > > > > probably going to make it due to lack of schema.
> > > > >
> > > > > So, here I am, calling for your experience! Hurl ins(igh|ul)ts
> > > at
> > > > > me,
> > > > > preferable here on this list.
> > > > >
> > > > > Best regards,
> > > > > Marcus
> > > > >
> > > > >
> > > > > [1] https://capnproto.org/index.html
> > > > >
> > > > > ¹This doesn't come as a surprise – it's just a good point in
> > > time
> > > > > to
> > > > > point this out.
> > > > > ²You've probably noticed that we're working towards a less
> > > machine-
> > > > > dependent, more portable, distributable computational model
> > > > > ³This can become problematic for messages with large data blobs
> > > in
> > > > > between, or very complex protocol headers
> > > > > ⁴CloudFlare seems to be using Cap'n Proto for Web workers, for
> > > > > example.
> > > > > However, it's but at revision 0.7, which indeed implies a bit
> > > of
> > > > > headache – other tools that never reached 1.0 that we rely on
> > > > > include
> > > > > Thrift, and that's really not a club that you'd want to be in.
> > > > >
> > > > >
> > > > > _______________________________________________
> > > > > Discuss-gnuradio mailing list
> > > > > Discuss-gnuradio@gnu.org
> > > > > https://lists.gnu.org/mailman/listinfo/discuss-gnuradio
> > >
> > >
> > > _______________________________________________
> > > Discuss-gnuradio mailing list
> > > Discuss-gnuradio@gnu.org
> > > https://lists.gnu.org/mailman/listinfo/discuss-gnuradio
> >
> > _______________________________________________
> > Discuss-gnuradio mailing list
> > Discuss-gnuradio@gnu.org
> > https://lists.gnu.org/mailman/listinfo/discuss-gnuradio
>
>
_______________________________________________
Discuss-gnuradio mailing list
Discuss-gnuradio@gnu.org
https://lists.gnu.org/mailman/listinfo/discuss-gnuradio
