Re: Question on Exactness of Arrow Memory Layout

2016-06-02 Thread Jacques Nadeau
What we're ultimately targeting is that the flatbuffer pointers Micah noted in [1] above can work a non-contiguous memory region. Take a look here at [2]. Each of the colored boxes should be contiguous, but they don't need to be packed together in memory for IPC. Note that the "data header" in [2]

Re: Question on Exactness of Arrow Memory Layout

2016-06-01 Thread Micah Kornfield
Hi Jacob, The current rough prototype/proposal of the IPC via shared memory is to do a depth first traversal of each arrays buffer and write them out to a contiguous memory block. Metadata about array types and locations of buffers is persisted at the end of memory block in flatbuffer format [1].