Hi Francois -- copying the metadata into memory isn't the end of the world but it's a pretty ugly wart. This affects every IPC protocol message everywhere.
We have an opportunity to address the wart now but such a fix post-1.0.0 will be much more difficult. On Thu, Jul 11, 2019, 2:05 PM Francois Saint-Jacques < fsaintjacq...@gmail.com> wrote: > If the data buffers are still aligned, then I don't think we should > add a breaking change just for avoiding the copy on the metadata? I'd > expect said metadata to be small enough that zero-copy doesn't really > affect performance. > > François > > On Sun, Jun 30, 2019 at 4:01 AM Micah Kornfield <emkornfi...@gmail.com> > wrote: > > > > While working on trying to fix undefined behavior for unaligned memory > > accesses [1], I ran into an issue with the IPC specification [2] which > > prevents us from ever achieving zero-copy memory mapping and having > aligned > > accesses (i.e. clean UBSan runs). > > > > Flatbuffer metadata needs 8-byte alignment to guarantee aligned accesses. > > > > In the IPC format we align each message to 8-byte boundaries. We then > > write a int32_t integer to to denote the size of flat buffer metadata, > > followed immediately by the flatbuffer metadata. This means the > > flatbuffer metadata will never be 8 byte aligned. > > > > Do people care? A simple fix would be to use int64_t instead of int32_t > > for length. However, any fix essentially breaks all previous client > > library versions or incurs a memory copy. > > > > [1] https://github.com/apache/arrow/pull/4757 > > [2] https://arrow.apache.org/docs/ipc.html >
