Hi Vitaly, My guess is that Python allocates byte buffers in a non-aligned way, e.g. maybe it always prefixes them with a 32-bit size. Probably you don't want to use to_bytes() here, but will need to introduce something lower-level that lets you get more control...
-Kenton On Mon, May 8, 2017 at 3:30 PM, vitaly numenta < [email protected]> wrote: > I was looking at this example from pycapnp that appears to bypass the > alignment-related copy whenever possible: > > ``` > > @cython.internal > > cdef class _FlatArrayMessageReader(_MessageReader): > > cdef object _object_to_pin > > def __init__(self, buf, traversal_limit_in_words = None, nesting_limit > = None): > > cdef schema_cpp.ReaderOptions opts > > cdef _AlignedBuffer aligned > > > if traversal_limit_in_words is not None: > > opts.traversalLimitInWords = traversal_limit_in_words > > if nesting_limit is not None: > > opts.nestingLimit = nesting_limit > > > sz = len(buf) > > if sz % 8 != 0: > > raise ValueError("input length must be a multiple of eight > bytes") > > > cdef char * ptr = buf > > if (<uintptr_t>ptr) % 8 != 0: > > aligned = _AlignedBuffer(buf) > > ptr = aligned.buf > > self._object_to_pin = aligned > > else: > > self._object_to_pin = buf > > > self.thisptr = new schema_cpp.FlatArrayMessageReader(schema_ > cpp.WordArrayPtr(<schema_cpp.word*>ptr, sz//8)) > > ``` > > > and tried a similar optimization in my application. The buffer in my > application comes from the "builder.to_bytes()" call in pycapnp. What I > found in my own experiment is that the buffer is never aligned and > therefore always needs to be copied to an aligned heap-based buffer. I > would like to be able to take advantage of this type of optimization in my > own extension, but I don't see how, since pycapnp's "to_bytes" seems to > always return a misaligned data buffer (at least on OS X). This is what my > python extension code looks like. In this snippet, the argument pyBytes is > the value returned by pycapnp's "to_bytes()" and the debug output is always > "ZZZ NEEDS ALIGNMENT": > > > ``` > > template<class MessageType> > > static void initFromPyBytes( > > nupic::Serializable<MessageType>& obj, > > const PyObject* pyBytes) > > { > > #if !CAPNP_LITE > > const char* srcBytes = nullptr; > > Py_ssize_t srcNumBytes = 0; > > > // NOTE: srcBytes will be set to point to the internal buffer inside > > // pyRegionProtoBytes' > > PyString_AsStringAndSize(const_cast<PyObject*>(pyBytes), > > const_cast<char**>(&srcBytes), > > &srcNumBytes); > > > if (srcNumBytes % sizeof(capnp::word) != 0) > > { > > throw std::logic_error( > > "PyCapnpHelper.initFromPyBytes input length must be a multiple > of " > > "capnp::word."); > > } > > const int srcNumWords = srcNumBytes / sizeof(capnp::word); > > > // Ensure alignment on capnp::word boundary; copy only if we have to > > kj::Array<capnp::word> aligned; > > if ((reinterpret_cast<uintptr_t>(srcBytes) % sizeof(capnp::word)) > != 0) > > { > > NTA_DEBUG << "ZZZ NEEDS ALIGNMENT"; > > aligned = kj::heapArray<capnp::word>(srcNumWords); > > memcpy(aligned.asBytes().begin(), srcBytes, srcNumBytes); // copy > > srcBytes = reinterpret_cast<char*>(aligned.asBytes().begin()); > > } > > else > > { > > NTA_DEBUG << "ZZZ ALREADY ALIGNED"; > > } > > > kj::ArrayPtr<capnp::word> wordArrayPtr( > > reinterpret_cast<capnp::word*>(const_cast<char*>(srcBytes)), > > srcNumWords); > > capnp::FlatArrayMessageReader reader(wordArrayPtr); // copy ? > > typename MessageType::Reader proto = reader.getRoot<MessageType>(); > > obj.read(proto); > > ``` > > -- > You received this message because you are subscribed to the Google Groups > "Cap'n Proto" group. > To unsubscribe from this group and stop receiving emails from it, send an > email to [email protected]. > Visit this group at https://groups.google.com/group/capnproto. > -- You received this message because you are subscribed to the Google Groups "Cap'n Proto" group. To unsubscribe from this group and stop receiving emails from it, send an email to [email protected]. Visit this group at https://groups.google.com/group/capnproto.
