Absolutely. One of the designs of batches of records if they include no more than 1 << 16 records (and may hold substantially less). A typical set of operations will create large numbers of these batches structures and many of them may be ephemeral.
The goal of the data headers structure (which is what holds the pointers to the individual field level structures) is to abstract a particular language based object model. It is also well defined as an in-memory representation and two applications can theoretically agree that these headers can be updated in place. On Wed, Feb 24, 2016 at 11:29 AM, Corey Nolet <cjno...@gmail.com> wrote: > Thanks for the fast response Jacques. But that's not to say it would be > impossible to continue to create new data structures and de-allocate others > in order to add and remove data, correct? > > On Wed, Feb 24, 2016 at 2:26 PM, Jacques Nadeau <jacq...@apache.org> > wrote: > > > Both are possible. Arrow doesn't force one or the other. You can copy > > between memory spaces and you still benefit from the same representation > > (avoiding ser/deserialization). However, the target is definitely > > leveraging shared memory. > > > > The way memory spaces are shared still needs more definition. My > > expectation is that in most cases the consumer of the structures will > > provide memory for purposes of population by the provider. That being > said, > > we'll need to flip this in the case of shared in-memory arrow caching > > layers as those get created. > > > > Back to the original question about mutability: > > > > These structures are designed to be write once, read many. So once they > are > > constructed, they are immutable. > > > > > > > > > > On Wed, Feb 24, 2016 at 11:17 AM, Andrew Brust < > > andrew.br...@bluebadgeinsights.com> wrote: > > > > > Hmm...that's not exactly how Jaques described things to me when he > > briefed > > > me on Arrow ahead of the announcement. > > > > > > -----Original Message----- > > > From: Zhe Zhang [mailto:z...@apache.org] > > > Sent: Wednesday, February 24, 2016 2:08 PM > > > To: dev@arrow.apache.org > > > Subject: Re: Question about mutability > > > > > > I don't think one application/process's memory space will be made > > > available to other applications/processes. It's fundamentally hard for > > > processes to share their address spaces. > > > > > > IIUC, with Arrow, when application A shares data with application B, > the > > > data is still duplicated in the memory spaces of A and B. It's just > that > > > data serialization/deserialization are much faster with Arrow (compared > > > with Protobuf). > > > > > > On Wed, Feb 24, 2016 at 10:40 AM Corey Nolet <cjno...@gmail.com> > wrote: > > > > > > > Forgive me if this question seems ill-informed. I just started > looking > > > > at Arrow yesterday. I looked around the github a tad. > > > > > > > > Are you expecting the memory space held by one application to be > > > > mutable by that application and made available to all applications > > > > trying to read the memory space? > > > > > > > > > >
