To follow up the discussion from the bi-weekly Arrow sync:
- JSON seems the most suitable candidate for the extension metadata.
E.g.: TensorArray
{"key": "ARROW:extension:name", "value": "tensor<type=int64, shape=(3,
3, 4), strides=(12, 4, 1)>"},
{"key": "ARROW:extension:metadata", "value": "{'type': 'int64',
'shape': [3, 3, 4], 'strides': [12, 4, 1]}"}
- We want to start with at least one integration test pair. Potential
candidates are cpp, julia, go, rust.
- First well known extension type candidate is TensorArray but other
suggestions are welcome.
On Tue, Jan 25, 2022 at 10:34 AM Antoine Pitrou <anto...@python.org> wrote:
>
>
> Le 25/01/2022 à 10:12, Joris Van den Bossche a écrit :
> > On Sat, 22 Jan 2022 at 20:27, Rok Mihevc <rok.mih...@gmail.com> wrote:
> >>
> >> Thanks for the input Weston!
> >>
> >> How about arrow/experimental/format/ExtensionTypes.fbs or
> >> arrow/format/ExtensionTypes.fbs for language independent schema and
> >> loosely arrow/<IMPLEMENTATION>/extensions for implementations?
> >>
> >> Having machine readable definitions could perhaps be useful for
> >> generating implementations in some cases.
> >
> > Is it useful to put this in a flatbuffer file? Based on the list from
> > Weston just below, I think this will mostly contain a *description* of
> > those different aspect (a specification of the extension type), and
> > there is no data that actually fits in a flatbuffer table? In that
> > case a plain text (eg markdown) file seems more fitting?
>
> I agree this is mostly a plain text (or, rather, reST :-)) specification
> task.
>
> Regards
>
> Antoine.
