+1, sounds ok to me to try to solve this problem a different way in the future once needed.
On Thu, Oct 19, 2017 at 12:30 PM, Jacques Nadeau <jacq...@apache.org> wrote: > Seems reasonable. I was among those that originally argued for this field > but given that we haven't used it yet, I think your proposal makes sense. > > +1 > > On Wed, Oct 18, 2017 at 5:40 PM, Wes McKinney <wesmck...@gmail.com> wrote: > > > When we originally drafted the metadata for record batches, we > > included a "page id" in the Buffer struct: > > > > https://github.com/apache/arrow/blob/master/format/Schema.fbs#L295 > > > > The idea at the time was that record batches might not be colocated in > > a particular shared memory page. This might still happen in the > > future, but to this point we have not used this feature in any > > implemented. > > > > The cost of this extra 4 bytes is that the size of the Buffer struct > > with padding is 24 bytes instead of 16 bytes. In large record batches, > > this makes the record batch data header about 50% larger than it needs > > to be. > > > > I would argue that the ability to spread a record batch across > > multiple memory regions is a useful feature, but we should be solving > > that particular problem a different way, like having a separate > > "non-colocated buffer" type and record batch message type that has the > > extra page id. So when we want to use this feature, we are OK with > > paying the extra cost. But for most self-contained message use cases > > those 8 bytes in each buffer go unused. > > > > I am loathe to break the Arrow metadata at this stage, but if we agree > > about removing this field we should do it sooner rather than later. It > > may be possible to do the change in a forward compatible way if we > > were worried about breaking existing applications, but on the other > > hand I do not think we have yet made any contract about > > forward/backwards compatibility of metadata with our end users. > > > > Thanks, > > Wes > > >
