Kenton, thanks for helping bring some clarity to this. It sounds like our two options are:
1. Require pycapnp and our extensions to be compiled in the same environment. We could potentially do this. We could make the install process easy for end users by forking pycapnp and putting wheels up on PyPI but we'd like to avoid that if possible. 2. Pass the byte buffers, incurring a memory copy for anything that we pass across the boundary. I'd like to explore #2 a bit more. Would this involve extracting the segments from the pycapnp builder/reader, passing that to our extension, and constructing a new builder/reader around the byte buffer? Or would we have to construct a new message in the extension, pass the segments from that back and find a way to copy that buffer into the pycapnp message builder/reader? We are also happy to put together a little demo project once we figure this out so others that want to do something similar have a starting place. On Wednesday, February 15, 2017 at 2:59:56 PM UTC-8, Kenton Varda wrote: > > Hi Vitaly, > > For ABI compatibility, you'd need pycapnp built against exactly the same > version of Cap'n Proto which you're using elsewhere in the process. Ideally > both would link against the same libcapnp.so, although I *think* loading > two copies of the library should not create problems as long as they are > the same version. (This differs from libprotobuf, which definitely can't > handle being loaded multiple times in the same process.) > > You may also need to make sure both copies are built with the same > compiler. We're aware of at least one ABI incompatibility issue between > Clang and GCC that affects Cap'n Proto. > > Of course, if you can't make anything work, you can always fall back to > transferring byte buffers, at the expense of possibly needing to make a > copy to merge the sub-messages into one overall message. > > -Kenton > > On Tue, Feb 14, 2017 at 4:46 PM, vitaly numenta <[email protected] > <javascript:>> wrote: > >> I am experiencing binary compatibility issues trying to get pycapnp >> serialization/deserialization working with C extensions. There appear to be >> ABI compatibility issues when passing C++ structs compiled in pycapnp into >> our C extensions that are compiled in a different environment. >> >> When serializing an instance of a class that's implemented in NuPIC, we >> create a message builder via pycapnp and pass it to the corresponding >> instance's write method, which in turn invokes write methods of its own >> contained members. This works fine for members whose classes are >> implemented in python, but doesn't always work for those implemented in the >> nupic.bindings extension due to ABI issues. >> >> For example, when serializing the TemporalMemory class, we might employ >> the following sequence: >> >> from nupic.proto import TemporalMemoryProto_capnp >> >> builder = TemporalMemoryProto_capnp.TemporalMemoryProto.new_message() >> >> temporal_memory.write(builder) >> >> Inside TemporalMemory.write(builder), we have something along these >> lines: >> >> class TemporalMemory(object): >> def write(self, builder): >> builder.columnDimensions = list(self.columnDimensions) >> self.connections.write(builder.connections) # pure python >> *self._random.write(builder.random) # C++ Random class from >> extension* >> >> >> The Random class that's implemented inside the nupic.bindings extension >> needs to rely on our own build of capnproto that's linked into the >> extension, but this doesn't seem to be compatible with the object >> constructed in pycapnp. >> >> We learned the hard way, after much trial and error, that we can't simply >> pass the underlying message builders that were instantiated by pycapnp's >> capnp.so module to our own build of capnproto contained in the >> nupic.bindigns extension. This was particularly evident when working on the >> manylinux wheel for nupic.bindings, which needs to be compiled using the >> toolchain and c/c++ runtimes from CentOS-6. This resulted in ABI >> incompatibilities when the capnproto code compiled into the extension >> attempts to operate on a message builder that was constructed by pycapnp's >> build of capnp.so. The message builder instance created by pycapnp's >> capnp.so appears corrupted when operated upon by the capnproto code linked >> into the extension. >> >> >> Is there any recommendation for handling this dual Python/C-extension >> scenario that avoids the ABI compatibility problem with C++ objects? >> >> -- >> You received this message because you are subscribed to the Google Groups >> "Cap'n Proto" group. >> To unsubscribe from this group and stop receiving emails from it, send an >> email to [email protected] <javascript:>. >> Visit this group at https://groups.google.com/group/capnproto. >> > > -- You received this message because you are subscribed to the Google Groups "Cap'n Proto" group. To unsubscribe from this group and stop receiving emails from it, send an email to [email protected]. Visit this group at https://groups.google.com/group/capnproto.
