On Thu, 8 Apr 2021 at 19:52, Micah Kornfield <emkornfi...@gmail.com> wrote:
> 1. Do the standard libraries handle that metadata key automatically ? > > C++, Python and Java have facilities to support them automatically > (extensions needs to register themselves), I'm not sure about other > languages. > > 2. Are there other standard or best practice metadata keys that either > people generally use or the standard libraries expect? > > I think there are some things related to Pandas types Indeed, a "pandas" key is used for this. A description of the information stored can be found at https://pandas.pydata.org/docs/dev/development/developer.html#storing-pandas-dataframe-objects-in-apache-parquet-format > and parquet data that > are propagated automatically through metadata. For Parquet, we store the schema in an "ARROW:schema" key in the Parquet metadata. When reading back, that key will typically be dropped and not included in the resulting Arrow table's schema metadata, though. > Rust was looking into > including file sources but I'm not sure that is being externalized. There > might be other things, hopefully people will chime in. > > On Thu, Apr 8, 2021 at 10:47 AM Michael Lavina <michael.lav...@factset.com > > > wrote: > > > Hey Team, > > > > I noticed that under extension type the Arrow docs specifically call out > > to `ARROW:extension:name` and `ARROW:extension:metadata` as > > recommended/reserved metadata keys to handle extension types. > > > > 2 quick questions > > > > > > 1. Do the standard libraries handle that metadata key automatically ? > > 2. Are there other standard or best practice metadata keys that either > > people generally use or the standard libraries expect? > > > > Thank you, > > Michael > > >
