On Thu, Feb 16, 2017 at 2:29 PM, Kenton Varda <[email protected]> wrote:
> On Thu, Feb 16, 2017 at 12:51 PM, Scott Purdy <[email protected]> wrote: > >> Kenton, thanks for helping bring some clarity to this. It sounds like our >> two options are: >> >> 1. Require pycapnp and our extensions to be compiled in the same >> environment. We could potentially do this. We could make the install >> process easy for end users by forking pycapnp and putting wheels up on PyPI >> but we'd like to avoid that if possible. >> > > I would argue that pycapnp should somehow export its version of libcapnp > so that other Python extensions that also use libcapnp are able to reuse > the same one. It makes sense for any Python extension that uses libcapnp.so > to declare a dependency on pycapnp, I would think. > > But I have no idea what this looks like logistically. > I think it would be great to have pycapnp export a pure-C interface. Numpy, for instance, has Python function call that returns the paths needed to enable you to build extensions against numpy. But I'm pretty sure this has to be pure-C to avoid ABI issues if you don't want to enforce that everything is built which the exact same toolchain. Do you think that is possible with capnp? My understanding was that a C interface wasn't on the roadmap but perhaps a more limited interface for this specific use case wouldn't be quite as much work. > 2. Pass the byte buffers, incurring a memory copy for anything that we >> pass across the boundary. >> >> I'd like to explore #2 a bit more. Would this involve extracting the >> segments from the pycapnp builder/reader, passing that to our extension, >> and constructing a new builder/reader around the byte buffer? Or would we >> have to construct a new message in the extension, pass the segments from >> that back and find a way to copy that buffer into the pycapnp message >> builder/reader? >> > > There's no good way to share builders, since there would be no way for > them to synchronize memory allocation. So, once a buffer has been passed, > it needs to be read-only. > > If you are trying to build a message in Python code but have one branch of > the message be built in C++ code, I think what you'll need to do is create > a brand new MessageBuilder in C++, build just the C++ branch of the message > there, and then pass this message to Python. In Python, you could read the > message with a MessageReader and then copy the contents into the branch of > the final message. This is where the copy is incurred -- when moving data > from one message into another message. Presumably you can transmit > individual messages between languages without any copies. > > -Kenton > > >> We are also happy to put together a little demo project once we figure >> this out so others that want to do something similar have a starting place. >> >> On Wednesday, February 15, 2017 at 2:59:56 PM UTC-8, Kenton Varda wrote: >>> >>> Hi Vitaly, >>> >>> For ABI compatibility, you'd need pycapnp built against exactly the same >>> version of Cap'n Proto which you're using elsewhere in the process. Ideally >>> both would link against the same libcapnp.so, although I *think* loading >>> two copies of the library should not create problems as long as they are >>> the same version. (This differs from libprotobuf, which definitely can't >>> handle being loaded multiple times in the same process.) >>> >>> You may also need to make sure both copies are built with the same >>> compiler. We're aware of at least one ABI incompatibility issue between >>> Clang and GCC that affects Cap'n Proto. >>> >>> Of course, if you can't make anything work, you can always fall back to >>> transferring byte buffers, at the expense of possibly needing to make a >>> copy to merge the sub-messages into one overall message. >>> >>> -Kenton >>> >>> On Tue, Feb 14, 2017 at 4:46 PM, vitaly numenta <[email protected] >>> > wrote: >>> >>>> I am experiencing binary compatibility issues trying to get pycapnp >>>> serialization/deserialization working with C extensions. There appear to be >>>> ABI compatibility issues when passing C++ structs compiled in pycapnp into >>>> our C extensions that are compiled in a different environment. >>>> >>>> When serializing an instance of a class that's implemented in NuPIC, we >>>> create a message builder via pycapnp and pass it to the corresponding >>>> instance's write method, which in turn invokes write methods of its >>>> own contained members. This works fine for members whose classes are >>>> implemented in python, but doesn't always work for those implemented in the >>>> nupic.bindings extension due to ABI issues. >>>> >>>> For example, when serializing the TemporalMemory class, we might employ >>>> the following sequence: >>>> >>>> from nupic.proto import TemporalMemoryProto_capnp >>>> >>>> builder = TemporalMemoryProto_capnp.TemporalMemoryProto.new_message() >>>> >>>> temporal_memory.write(builder) >>>> >>>> Inside TemporalMemory.write(builder), we have something along these >>>> lines: >>>> >>>> class TemporalMemory(object): >>>> def write(self, builder): >>>> builder.columnDimensions = list(self.columnDimensions) >>>> self.connections.write(builder.connections) # pure python >>>> *self._random.write(builder.random) # C++ Random class from >>>> extension* >>>> >>>> >>>> The Random class that's implemented inside the nupic.bindings extension >>>> needs to rely on our own build of capnproto that's linked into the >>>> extension, but this doesn't seem to be compatible with the object >>>> constructed in pycapnp. >>>> >>>> We learned the hard way, after much trial and error, that we can't >>>> simply pass the underlying message builders that were instantiated by >>>> pycapnp's capnp.so module to our own build of capnproto contained in the >>>> nupic.bindigns extension. This was particularly evident when working on the >>>> manylinux wheel for nupic.bindings, which needs to be compiled using the >>>> toolchain and c/c++ runtimes from CentOS-6. This resulted in ABI >>>> incompatibilities when the capnproto code compiled into the extension >>>> attempts to operate on a message builder that was constructed by pycapnp's >>>> build of capnp.so. The message builder instance created by pycapnp's >>>> capnp.so appears corrupted when operated upon by the capnproto code linked >>>> into the extension. >>>> >>>> >>>> Is there any recommendation for handling this dual Python/C-extension >>>> scenario that avoids the ABI compatibility problem with C++ objects? >>>> >>>> -- >>>> You received this message because you are subscribed to the Google >>>> Groups "Cap'n Proto" group. >>>> To unsubscribe from this group and stop receiving emails from it, send >>>> an email to [email protected]. >>>> Visit this group at https://groups.google.com/group/capnproto. >>>> >>> >>> -- >> You received this message because you are subscribed to the Google Groups >> "Cap'n Proto" group. >> To unsubscribe from this group and stop receiving emails from it, send an >> email to [email protected]. >> Visit this group at https://groups.google.com/group/capnproto. >> > > -- > You received this message because you are subscribed to a topic in the > Google Groups "Cap'n Proto" group. > To unsubscribe from this topic, visit https://groups.google.com/d/ > topic/capnproto/MG9RijMCpHo/unsubscribe. > To unsubscribe from this group and all its topics, send an email to > [email protected]. > Visit this group at https://groups.google.com/group/capnproto. > -- You received this message because you are subscribed to the Google Groups "Cap'n Proto" group. To unsubscribe from this group and stop receiving emails from it, send an email to [email protected]. Visit this group at https://groups.google.com/group/capnproto.
