Hi Stefan,
On Fri, May 4, 2018 at 11:08 AM, <[email protected]> wrote:
> Hello!
>
> I'm interested in using Cap'n Proto for serializing a large-ish quad tree
> data structure that is used for querying geospatial data. I have a few
> questions that I was hoping you could help me out with:
>
> - Have you come across any message size limitations or performance
> issues with larger data in Cap'n Proto? I'd love to be able to represent
> indices on the order of 100 MB+.
>
> As long as you are mmap()ing in the file, the total size of the file will
have no bearing on performance -- the only thing that matters is how much
of the object tree you actually traverse.
Note that mmap() doesn't allow for any userspace compression. If you find
you're spending a lot of time in disk I/O, you may want to enable
compression at the filesystem level.
>
> - In a post made on 8/1/14 ("Recursive Schemas"), you mentioned that
> there can only be a single pointer to other structs -- is this still the
> case? I would love to be able to have a pointer to parent nodes so that I
> can traverse up through the quad tree.
>
> Sorry, but capnp is still a tree structure, not a graph. You'll need to
remember parent node pointers on a stack as you traverse.
>
> - How are structs laid out in memory (via arena allocation) when
> working with nested structs? Is it based on the order of "init" statements?
> I'd like to maintain a breadth-first layout of nested structs in my
> serialized output to maintain locality of nodes at a specific depth -- do I
> just need to initialize the structs in the order at which I want them laid
> out in the serialized representation?
>
> Yes, they will be ordered in memory in the order in which they were
allocated (which happens when you call "init").
Note you may want to tune the constructor parameters to
MallocMessageBuilder to make sure you are allocating large segments, to
avoid fragmentation near the start of the message. Or you may want to
implement your own MessageBuilder subclass.
Are there any other considerations I should take into account?
>
I'm assuming this is a data structure that you'll build once, and then use
many times without modifying it further. If so, it should work well. If you
need to continuously modify, Cap'n Proto isn't very good at that right now.
-Kenton
--
You received this message because you are subscribed to the Google Groups
"Cap'n Proto" group.
To unsubscribe from this group and stop receiving emails from it, send an email
to [email protected].
Visit this group at https://groups.google.com/group/capnproto.
