Thanks for the input Weston!
How about arrow/experimental/format/ExtensionTypes.fbs or
arrow/format/ExtensionTypes.fbs for language independent schema and
loosely arrow/<IMPLEMENTATION>/extensions for implementations?
Having machine readable definitions could perhaps be useful for
generating implementations in some cases.
> * The name of the extension type (to go in ARROW:extension:name)
> * A description of the extension type and how it should be used
> * The storage type of the extension type
> * The format and meaning of the content that will go into
> ARROW:extension:metadata
These sound pretty complete!
I'll wait for a couple of days to see if there's more input and then
draft a PR. Do we need a vote on this?
Best,
Rok
On Fri, Jan 21, 2022 at 3:07 AM Weston Pace <weston.p...@gmail.com> wrote:
>
> Those all seem to be C++ locations. If we want to define
> cross-implementation "Well Known Extension Types" then it seems we
> would want to come up with some kind of language independent agreement
> (could just be a markdown file but maybe there is some advantage to
> having something programmatically consumable) describing:
>
> * The name of the extension type (to go in ARROW:extension:name)
> * A description of the extension type and how it should be used
> * The storage type of the extension type
> * The format and meaning of the content that will go into
> ARROW:extension:metadata
>
> I think (but am not sure) that, since these are metadata keys, we are
> supposed to stick to printable ASCII for values (for backwards
> compatibility).
>
> For example, in the docs, we currently have this little blurb about a
> theoretical tensor extension type:
>
> > tensor (multidimensional array) stored as Binary values and
> > having serialized metadata indicating the data type and shape
> > of each value. This could be JSON like {'type': 'int8', 'shape':
> > [4, 5]} for a 4x5 cell tensor.
>
> In my mind this file would be somewhat analogous to the way that
> schema.fbs is the cross implementation "ground truth" for our logical
> types.
>
> Then the C++ implementation would be free to put the implementation
> (I'd vote for arrow/cpp/extensions but a separate repo is probably ok.
> I'm -1 on arrow/extensions/...)
>
> On Thu, Jan 20, 2022 at 3:20 PM Rok Mihevc <rok.mih...@gmail.com> wrote:
> >
> > To continue the ExtensionType part of this thread - I would like to
> > add TensorArray [1] as an ExtensionType to Arrow but we have not yet
> > agreed on an "official" location for "Well Known Extension Types".
> >
> > Where could we put these? Some suggestions:
> >
> > * implementation folders (e.g. arrow/cpp/extensions/tensor_array.h)
> > * extensions folder (e.g. arrow/extensions/cpp/tensor_array.h)
> > * separate repo (e.g. github.com/apache/arrow_extensions/cpp/tensor_array.h)
> >
> > I'd be happy to also gather other Well Known Extension Types into one
> > location if this moves forward.
> >
> > Rok
> >
> > [1] https://github.com/apache/arrow/pull/8510#issuecomment-991150389
> >
> > On Sat, May 1, 2021 at 12:12 PM Andrew Lamb <al...@influxdata.com> wrote:
> > >
> > > I agree with others on this thread. Thanks for writing this down Micah
> > >
> > > On Fri, Apr 30, 2021 at 11:16 AM Antoine Pitrou <anto...@python.org>
> > > wrote:
> > >
> > > >
> > > > I concur with both what Wes and Micah said.
> > > >
> > > > As for temporal types, they have wide-spread use and their semantics
> > > > require dedicated treatment for arithmetic and conversion, so it's
> > > > helpful to define dedicated types for them, as opposed to mere
> > > > annotations.
> > > >
> > > > Regards
> > > >
> > > > Antoine.
> > > >
> > > >
> > > > Le 30/04/2021 à 16:40, Wes McKinney a écrit :
> > > > > I agree that the bar for adding new types to the Type union in
> > > > > Schema.fbs
> > > > > should be quite high going forward. Using extension types increasingly
> > > > for
> > > > > adding specializations of built-in types will mean less burden for
> > > > > implementations to simply "propagate forward" this data (by preserving
> > > > the
> > > > > extra metadata) even if they don't understand what it does. It would
> > > > > be
> > > > > nice, therefore, to put us on a path to expanding our set of
> > > > > "official"
> > > > > extension types (which would include things like JSON or UUID) since
> > > > > some
> > > > > libraries may choose to implement convenience containers for these for
> > > > > usability.
> > > > >
> > > > > On Fri, Apr 30, 2021 at 9:22 AM Brian Hulette <bhule...@apache.org>
> > > > wrote:
> > > > >
> > > > >> +1 this looks good to me.
> > > > >>
> > > > >> My only concern is with criteria #3 " Is the underlying encoding of
> > > > >> the
> > > > >> type already semantically supported by a type?". I think this is a
> > > > >> good
> > > > >> criteria, but it's inconsistent with the current spec. By that
> > > > >> criteria
> > > > >> some existing types (Timestamp, Time, Duration, Date) should be well
> > > > known
> > > > >> extension types, right?
> > > > >>
> > > > >> Perhaps we should explicitly indicate these types are grandfathered
> > > > >> in
> > > > [1]
> > > > >> because they existed before extension types, to avoid tension with
> > > > >> this
> > > > >> criteria.
> > > > >>
> > > > >> Brian
> > > > >>
> > > > >> [1] https://en.wikipedia.org/wiki/Grandfather_clause
> > > > >>
> > > > >> On Thu, Apr 29, 2021 at 9:13 PM Jorge Cardoso Leitão <
> > > > >> jorgecarlei...@gmail.com> wrote:
> > > > >>
> > > > >>> Thanks for writing this.
> > > > >>>
> > > > >>> I agree. That is a good decision tree. +1
> > > > >>>
> > > > >>> Best,
> > > > >>> Jorge
> > > > >>>
> > > > >>>
> > > > >>> On Thu, Apr 29, 2021 at 6:08 PM Micah Kornfield
> > > > >>> <emkornfi...@gmail.com
> > > > >
> > > > >>> wrote:
> > > > >>>
> > > > >>>> The discussion around adding another interval type to the
> > > > >>>> Schema.fbs
> > > > >>> raises
> > > > >>>> the issue of when do we decide to add a new type to the Schema.fbs
> > > > >>>> vs
> > > > >>> using
> > > > >>>> other means (primarily extension types [1]).
> > > > >>>>
> > > > >>>> A few criteria come to mind that could help decide (feedback
> > > > >>>> welcome):
> > > > >>>>
> > > > >>>> 1. Is the type a new parameterization of an existing type?
> > > > >>>> - If Yes, and we believe the parameterization is useful and
> > > > >>>> can
> > > > be
> > > > >>> done
> > > > >>>> in a forward/backward compatible manner then we would update
> > > > >> Schema.fbs.
> > > > >>>>
> > > > >>>> 2. Does the type itself have its own specification for processing
> > > > >> (e.g.
> > > > >>>> JSON, BSON, Thrift, Avro, Protobuf)?
> > > > >>>> - If yes, we would NOT add them to Schema.fbs. I think this
> > > > >>>> would
> > > > >>>> potentially yield too many new types.
> > > > >>>>
> > > > >>>> 3. Is the underlying encoding of the type already semantically
> > > > >> supported
> > > > >>>> by a type? (e.g. if we want to encode physical lengths like meters
> > > > >> these
> > > > >>>> can be represented by an integer).
> > > > >>>> - If yes, we would NOT update the specification. This seems
> > > > >>>> like
> > > > >> the
> > > > >>>> exact use-case that extension types are meant for.
> > > > >>>>
> > > > >>>> * How does this apply to Interval? *
> > > > >>>> Interval extends an existing type in the specification and multiple
> > > > >>> "packed
> > > > >>>> fields" cannot be easily communicated with the current version of
> > > > >>>> the
> > > > >>>> specification. Hence, I feel comfortable making the addition to
> > > > >>> Schema.fbs
> > > > >>>>
> > > > >>>> * What does this mean for other common types? *
> > > > >>>>
> > > > >>>> I think as types come up that are very common but we don't want to
> > > > >>>> add
> > > > >> to
> > > > >>>> the Schema.fbs we should invest in formalizing them as "Well Known"
> > > > >>>> Extension types. In this scenario, we would update the
> > > > >>>> specification
> > > > >> to
> > > > >>>> include how to specify the extension type metadata (and still
> > > > >>>> require
> > > > >> at
> > > > >>>> least two libraries support the Extension type before inclusion as
> > > > >> "Well
> > > > >>>> Known").
> > > > >>>>
> > > > >>>> * Practical implications *
> > > > >>>>
> > > > >>>> I think this means the type system in Schema.fbs is mostly closed
> > > > (i.e.
> > > > >>>> there is a high bar for adding new types). One potentially useful
> > > > >>>> type
> > > > >> to
> > > > >>>> have would be a "packed struct" that supports something similar to
> > > > >> python
> > > > >>>> struct library [2]. I think this would likely cover many extension
> > > > >> type
> > > > >>>> use-cases.
> > > > >>>>
> > > > >>>> Thoughts?
> > > > >>>>
> > > > >>>> -Micah
> > > > >>>>
> > > > >>>> [1]
> > > > https://arrow.apache.org/docs/format/Columnar.html#extension-types
> > > > >>>> [2] https://docs.python.org/3/library/struct.html
> > > > >>>>
> > > > >>>
> > > > >>
> > > > >
> > > >
