> Is my understanding correct? I believe your understanding is correct.
> Are there additional considerations in rust which means it's not possible for objects accessed by application code (e.g. record-batches, arrays) to keep their buffers in IPC messages? I don't think there is anything specific in Rust that prevents zero copy of the IPC buffer. I think what is missing is someone to sit down with the IPC reader/writer code implementation and optimize -- e.g. perhaps create a shared `Bytes` directly from the IPC message data, and then create `Buffer`/ `ArrayData` pointing into that and build the final arrays On Tue, Apr 6, 2021 at 12:28 PM Vertexclique <vertexcli...@gmail.com> wrote: > Hi Al; > > Yeah, that's true. The ownership rules of Rust prevents resource ownership > outside the container that the resource lifetime taken. > > That's why there are buffer copies to take ownership and operate on it. > Imo, it should be also removed from the docs if it's written like that (for > Rust). > > Best, > Theo > > On Apr 6, 2021, 17:03, at 17:03, Al Taylor > <al.taylor1...@googlemail.com.invalid> > wrote: > >Hi, > > > >I was reading around the rust-arrow codebase, evaluating it for > >potential future use. I'm particularly interested in zero-copy > >processing. > > > >I could very well be wrong here, as I don't have a lot of rust > >experience, but it looks like the code for reading buffers out of IPC > >messages is copying the contents of the data. > > > >A buffer is constructed via Buffer::from(&[u8]) according to an > >ipc::Buffer object here: > > > https://github.com/apache/arrow/blob/8e43f23dcc6a9e630516228f110c48b64d13cec6/rust/arrow/src/ipc/reader.rs#L39 > > > >The Buffer::from method is defined here: > > > https://github.com/apache/arrow/blob/8e43f23dcc6a9e630516228f110c48b64d13cec6/rust/arrow/src/buffer/immutable.rs#L222 > >and appears to create an empty buffer, then copy data from the slice > >into it. > > > >As far as I'm aware, the C++ implementation of arrow does not copy > >buffer data out of IPC messages in this way. > > > >Is my understanding correct? Are there additional considerations in > >rust which means it's not possible for objects accessed by application > >code (e.g. record-batches, arrays) to keep their buffers in IPC > >messages? > > > >Thanks, > > > >Al >
