>
> We can't because the buffer layout is not transmitted -- implementations
> make assumptions about what Buffer values correspond to each field. The
> only thing we could do to signal the change would be to increase the
> metadata version from V4 to V5.
If we do this within 0.15.0 we could infer from the padding of messages.
On Fri, Sep 6, 2019 at 10:16 AM Wes McKinney <wesmck...@gmail.com> wrote:
> On Fri, Sep 6, 2019, 12:08 PM Antoine Pitrou <anto...@python.org> wrote:
>
> >
> > Null can also come up when converting a column with only NA values in a
> > CSV file. I don't remember for sure, but I think the same can happen
> > with JSON files as well.
> >
> > Can't we accept both forms when reading? It sounds like it should be
> > reasonably easy.
> >
>
> We can't because the buffer layout is not transmitted -- implementations
> make assumptions about what Buffer values correspond to each field. The
> only thing we could do to signal the change would be to increase the
> metadata version from V4 to V5.
>
>
> > Regards
> >
> > Antoine.
> >
> >
> > Le 06/09/2019 à 17:36, Wes McKinney a écrit :
> > > hi Micah,
> > >
> > > Null wouldn't come up that often in practice. It could happen when
> > > converting from pandas, for example
> > >
> > > In [8]: df = pd.DataFrame({'col1': np.array([np.nan] * 10,
> > dtype=object)})
> > >
> > > In [9]: t = pa.table(df)
> > >
> > > In [10]: t
> > > Out[10]:
> > > pyarrow.Table
> > > col1: null
> > > metadata
> > > --------
> > > {b'pandas': b'{"index_columns": [{"kind": "range", "name": null,
> > "start": 0, "'
> > > b'stop": 10, "step": 1}], "column_indexes": [{"name": null,
> > "field'
> > > b'_name": null, "pandas_type": "unicode", "numpy_type":
> > "object", '
> > > b'"metadata": {"encoding": "UTF-8"}}], "columns": [{"name":
> > "col1"'
> > > b', "field_name": "col1", "pandas_type": "empty",
> > "numpy_type": "o'
> > > b'bject", "metadata": null}], "creator": {"library":
> > "pyarrow", "v'
> > > b'ersion": "0.14.1.dev464+g40d08a751"}, "pandas_version":
> > "0.24.2"'
> > > b'}'}
> > >
> > > I'm inclined to make the change without worrying about backwards
> > > compatibility. If people have been persisting data against the
> > > recommendations of the project, the remedy is to use an older version
> > > of the library to read the files and write them to something else
> > > (like Parquet format) in the meantime.
> > >
> > > Obviously come 1.0.0 we'll begin to make compatibility guarantees so
> > > this will be less of an issue.
> > >
> > > - Wes
> > >
> > > On Thu, Sep 5, 2019 at 11:14 PM Micah Kornfield <emkornfi...@gmail.com
> >
> > wrote:
> > >>
> > >> Hi Wes and others,
> > >> I don't have a sense of where Null arrays get created in the existing
> > code
> > >> base?
> > >>
> > >> Also, do you think it is worth the effort make this backwards
> > compatible.
> > >> We could in theory tie the buffer count to having the continuation
> value
> > >> for alignment.
> > >>
> > >> The one area were I'm slightly concerned is we seem to have users in
> the
> > >> wild who are depending on backwards compatibility, and I'm try to
> better
> > >> understand the odds that we break them.
> > >>
> > >> Thanks,
> > >> Micah
> > >>
> > >> On Thu, Sep 5, 2019 at 7:25 AM Wes McKinney <wesmck...@gmail.com>
> > wrote:
> > >>
> > >>> hi folks,
> > >>>
> > >>> One of the as-yet-untested (in integration tests) parts of the
> > >>> columnar specification is the Null layout. In C++ we additionally
> > >>> implemented this by writing two length-0 "placeholder" buffers in the
> > >>> RecordBatch data header, but since the Null layout has no memory
> > >>> allocated nor any buffers in-memory it may be more proper to write no
> > >>> buffers (since the length of the Null layout is all you need to
> > >>> reconstruct it). There are 3 implementations of the placeholder
> > >>> version (C++, Go, JS, maybe also C#) but it never got implemented in
> > >>> Java. While technically this would break old serialized data, I would
> > >>> not expect this to be very frequently occurring in many of the
> > >>> currently-deployed Arrow applications
> > >>>
> > >>> Here is my C++ patch
> > >>>
> > >>> https://github.com/apache/arrow/pull/5287
> > >>>
> > >>> I'm not sure we need to formalize this with a vote but I'm interested
> > >>> in the community's feedback on how to proceed here.
> > >>>
> > >>> - Wes
> > >>>
> >
>
