Hi Wes and others, I don't have a sense of where Null arrays get created in the existing code base?
Also, do you think it is worth the effort make this backwards compatible. We could in theory tie the buffer count to having the continuation value for alignment. The one area were I'm slightly concerned is we seem to have users in the wild who are depending on backwards compatibility, and I'm try to better understand the odds that we break them. Thanks, Micah On Thu, Sep 5, 2019 at 7:25 AM Wes McKinney <wesmck...@gmail.com> wrote: > hi folks, > > One of the as-yet-untested (in integration tests) parts of the > columnar specification is the Null layout. In C++ we additionally > implemented this by writing two length-0 "placeholder" buffers in the > RecordBatch data header, but since the Null layout has no memory > allocated nor any buffers in-memory it may be more proper to write no > buffers (since the length of the Null layout is all you need to > reconstruct it). There are 3 implementations of the placeholder > version (C++, Go, JS, maybe also C#) but it never got implemented in > Java. While technically this would break old serialized data, I would > not expect this to be very frequently occurring in many of the > currently-deployed Arrow applications > > Here is my C++ patch > > https://github.com/apache/arrow/pull/5287 > > I'm not sure we need to formalize this with a vote but I'm interested > in the community's feedback on how to proceed here. > > - Wes >
