Hi Wylder, On Mon, Aug 8, 2022 at 12:16 PM Wylder Keane <wylder.ke...@gmail.com> wrote:
> Hey Kenton, > > Any chance the mutation friendly in-memory representation feature is on > the near-term roadmap? Seems like it would make using CapnProto even > easier. I've already implemented a wrapper object akin to what you referred > to (but using std::shared_ptr) and that works for immutable instances, but > it's still impossible to keep an object around and mutate it without > leaking memory every time a non-primitive field is set (which is not very > obvious from the documentation btw). > To be honest, there isn't really a roadmap for Cap'n Proto. I build the features I need when I need them, driven by my other projects. So far I haven't needed this feature. In the absence of a use case of my own it's sort of hard for me to have intuition about how this should be built. Like, what *kinds* of mutations should be supported efficiently? E.g. Lists should be efficiently appendable probably, but is efficient insertion also needed? When I think about this I tend to land on the conclusion that I would usually want to define mutable structures in my actual programming language and not in Cap'n Proto. Note that this hypothetical mutation-friendly representation would necessarily not be zero-copy to parse / serialize. Zero-copy and mutation-friendly just are not compatible design goals, unfortunately. > Related, is there any way to adopt all of the allocated segments from > another MallocMessageBuilder? Effectively, swap the two instances of > MallocMessageBuilder? > I'm not sure what you mean here -- adopting segments from one builder to another, vs. swapping them, seem like totally different things. Which is it? If you want a swap, could you allocate both MessageBuilders on the heap and swap the pointers? Or do you really want to be able to take one message and embed it within another, without copying? I have wanted that too, but it's difficult in part because far pointers refer to other segments by absolute index of the segment rather than relative, so you can't embed another message wholesale without fixing up the far pointers (unless your destination builder started out empty so that the segments could keep their same numbers). If I were to make one backwards-incompatible change to Cap'n Proto, this is what I'd change, incidentally. -Kenton > > Thanks, > Wylder > On Thursday, July 19, 2018 at 1:31:00 PM UTC-7 ken...@cloudflare.com > wrote: > >> Hi Stefan, >> >> Cap'n Proto's Readers and Builders both behave as pointers, not value >> types. They are copyable, but copying them doesn't copy the data, it just >> copies the pointer. Both actually point into the underlying message buffer, >> and calling methods on the Reader/Builder manipulates that underlying >> buffer. >> >> To actually move data from one builder to another, you'd have to do >> something like: >> >> // Copy primitive fields like this. >> to.setFoo(from.getFoo()); >> to.setBar(from.getBar()); >> >> // Transfer ownership of pointer fields like this. >> to.adoptBaz(from.disownBaz()); >> to.adoptQux(from.disownQux()); >> >> If you wanted to avoid manually specifying each field, you could write >> some code based on AnyStruct::Builder instead. Any struct builder type can >> be converted to AnyStruct::Builder, which then allows you to iterate over >> the raw data and pointers that make up the struct. >> >> Note that mutating Cap'n Proto messages is awkward due to the memory >> allocation needs, namely that all objects need to be allocated adjacent in >> the message buffer, to allow zero-copy output. Because of this, Cap'n Proto >> is not an ideal format for an in-memory data structure that you need to >> modify over time. The "self-contained structures in C++" thread you linked >> to, also knows as "POCS" (plain-old-c-structs), is an idea I'd still like >> to implement at some point, providing a mutation-friendly (but >> non-zero-copy) in-memory representation corresponding to Cap'n Proto types. >> No work has been done on this yet, unfortunately. >> >> -Kenton >> >> On Wed, Jul 18, 2018 at 10:27 AM, <stefan...@ai-solutions.com> wrote: >> >>> Hey Kenton, >>> >>> I came across the thread for Self-contained structures in C++ >>> <https://groups.google.com/forum/#!searchin/capnproto/refcounted%7Csort:date/capnproto/Reg0wInHBdY/ftJUP3iTCQAJ> >>> and it sounds like that what I'm needing to implement is a >>> refcounted-builder. That thread hasn't been updated in a couple of years, >>> so I was curious if that approach would still be valid. >>> >>> Thanks for your support! >>> >>> >>> On Tuesday, July 17, 2018 at 8:42:49 PM UTC-4, >>> stefan...@ai-solutions.com wrote: >>>> >>>> Hi Kenton, >>>> >>>> I have a struct which has a field containing a list of structs of the >>>> same type, akin to that of a tree structure. Is it possible to copy or >>>> obtain references for the struct Builder instances associated with a >>>> specific struct so that I can subsequently initialize lists for each struct >>>> in a breadth-first style? >>>> >>>> Initially, I attempted to store the message builder corresponding to >>>> the child nodes in a kj::Vector then std::move them to a parent nodes >>>> kj::Vector, hoping that I could initialize the children list in future >>>> iterations. However, I'm finding that when trying to reference a parent by >>>> its index, the message builder is instead references the current node's >>>> children (which just so happens to be the most recently instantiated >>>> instance of the message builder). >>>> >>>> Here's some example code that outlines what I'm trying to do: >>>> https://gist.github.com/slnovak/2f8b14845019da98566c6fc2776ee186 >>>> >>>> Thanks so much! >>>> >>> -- >>> You received this message because you are subscribed to the Google >>> Groups "Cap'n Proto" group. >>> To unsubscribe from this group and stop receiving emails from it, send >>> an email to capnproto+unsubscr...@googlegroups.com. >>> Visit this group at https://groups.google.com/group/capnproto. >>> >> >> -- > You received this message because you are subscribed to the Google Groups > "Cap'n Proto" group. > To unsubscribe from this group and stop receiving emails from it, send an > email to capnproto+unsubscr...@googlegroups.com. > To view this discussion on the web visit > https://groups.google.com/d/msgid/capnproto/0eb01b62-ac7f-44ef-9bc1-f46e799c5be8n%40googlegroups.com > <https://groups.google.com/d/msgid/capnproto/0eb01b62-ac7f-44ef-9bc1-f46e799c5be8n%40googlegroups.com?utm_medium=email&utm_source=footer> > . > -- You received this message because you are subscribed to the Google Groups "Cap'n Proto" group. To unsubscribe from this group and stop receiving emails from it, send an email to capnproto+unsubscr...@googlegroups.com. To view this discussion on the web visit https://groups.google.com/d/msgid/capnproto/CAJouXQnOsrm9kgqTVRP7fKuY4T79bAdT_KgviFYcJ6Enh-xHUw%40mail.gmail.com.
