IIUC, https://github.com/apache/parquet-format/pull/451 is a different story which tries to add a new logical type in Parquet to enable users to register their custom type. Here we just want to define a mapping between existing Parquet phyiscal/logical types and pre-defined Arrow extension types.
On Mon, Nov 9, 2026 at 7:13 PM Rok Mihevc <[email protected]> wrote: > For context - an approach to extension types was proposed to Parquet in the > past [1][2] but was met with opposition because it was somewhat duplicating > logical types and not really reducing the cost of specifying types > (consensus building). > > I like Weston's proposal too, but I am not sure the Parquet community wants > to deal with blessing individual extension types. What we could propose is > that Parquet specifies how Arrow-compatible extension types serialize into > Parquet metadata. This would enable third party extension types and if > those show enough potential Parquet might canonicalize the popular ones. I > have a vague plan to raise such a proposal soon. > > Regarding the ongoing vote in this thread and Mandy's proposal, should we > put it on hold until we agree on this or continue with the vote? > > [1] https://github.com/apache/parquet-format/pull/451 > [2] https://github.com/rok/parquet-format/pull/5 > > Best, > Rok > > On Mon, Nov 9, 2026 at 10:08 AM Zehua Zou <[email protected]> wrote: > > > Hello, > > > > Should we initiate a discussion on the Parquet mailing list? > > > > Regards > > > > Zehua Zou > > > > > On Nov 9, 2026, at 16:22, Antoine Pitrou <[email protected]> wrote: > > > > > > > > > Hello, > > > > > > I have roughly the same position as Weston here. > > > > > > Regards > > > > > > Antoine. > > > > > > > > > > > > Le 09/11/2026 à 06:04, Weston Pace a écrit : > > >>> Are you saying there should be super-project canonical extension > > types? :) > > >> No. I am suggesting the Parquet PMC publish an extension-types.md > > document > > >> in [1] that describes what the extension type definitions are. This > > would > > >> be exactly like [2]. This lets the Parquet team bless whatever > > >> unambiguous definition they like. Anyone wanting to consume custom > > Parquet > > >> types must use those definitions. The Parquet team is then in > complete > > >> control and doesn't need to seek the Arrow team's blessing. The Arrow > > team > > >> can then create a link back to that document in [3] in much the same > > way we > > >> do for GeoArrow. > > >> For example, the Substrait type system does not name all Arrow types. > > in > > >> Arrow, we publish [4] which describes Substrait extension types to map > > to > > >> the existing Arrow types. > > >> Or, as another example, Lance's blob type is an Arrow extension type > > >> defined in [5]. Consumers of Lance custom types consult the Lance > spec > > for > > >> those types' definitions. > > >>> Similarily, I was thinking of adding more canonical Parquet extension > > >>> types like GEOMETRY and GEOGRAPHY because I found it really tricky > > >>> when implementing Iceberg V3 types in iceberg-cpp where GeoArrow type > > >>> were not an option. > > >> If Iceberg has its own set of types (separate from Parquet and > GeoArrow) > > >> then Iceberg can also maintain an extension-types.md document right? > > >> [1] https://github.com/apache/parquet-format/ > > >> [2] https://github.com/geoarrow/geoarrow/blob/main/extension-types.md > > >> [3] > > >> > > > https://arrow.apache.org/docs/format/CanonicalExtensions.html#community-extension-types > > >> [4] > > >> > > > https://github.com/apache/arrow/blob/main/format/substrait/extension_types.yaml > > >> [5] > > >> > > > https://github.com/lance-format/lance/blob/main/docs/src/format/table/blob.md#table-schema > > >> On Sat, Nov 7, 2026 at 7:00 AM Rok Mihevc <[email protected]> > wrote: > > >>> Thanks for the discussion Weston and Gang! > > >>> > > >>>> I also somewhat disagree with this characterization. As long as the > > >>>> producer and consumer agree on the type then there is no need for > > every > > >>>> middleman (or even Arrow itself) to be aware of it. That's the > entire > > >>> goal > > >>>> of extension types. IPC streams, the C data interface, Arrow > flight, > > and > > >>>> language bindings should all respect existing metadata and should > > _not_ > > >>>> strip it out, even if they don't recognize it. > > >>> > > >>> This is a great point and it's ideally how things should work. Now > how > > do > > >>> we get there? Should Parquet readers that output Arrow map Parquet > > logical > > >>> types into "parquet.<logical_type>" extension types? If this requires > > >>> non-trivial logic, to make the mapping unambiguous, are we implicitly > > >>> specifying extension types at the conversion layer? > > >>> > > >>>> and so I will turn my vote into a +1 if needed for passage. > Otherwise > > >>> I'll > > >>>> leave it as a half vote to discourage Arrow taking on the mantle of > > >>>> blessing another format's types. > > >>> > > >>> Are you saying there should be super-project canonical extension > > types? :) > > >>> > > >>> > > >>> Best, > > >>> Rok > > >>> > > >>> On Sat, Nov 7, 2026 at 2:57 AM Gang Wu <[email protected]> wrote: > > >>> > > >>>> Thanks Weston for raising this! I have a different opinion. > > >>>> > > >>>> Canonical extension type for Parquet types like VARIANT、FILE and > > future > > >>>> VECTOR brings better interoperability to different Parquet bindings > in > > >>>> Arrow. AFAIK, it would be a headache if different bindings define > > their > > >>>> own custom extensions then we end up with different interpretations > of > > >>>> ARROW:schema serialization across bindings. > > >>>> > > >>>> Similarily, I was thinking of adding more canonical Parquet > extension > > >>>> types like GEOMETRY and GEOGRAPHY because I found it really tricky > > >>>> when implementing Iceberg V3 types in iceberg-cpp where GeoArrow > type > > >>>> were not an option. > > >>>> > > >>>> On Sat, Nov 7, 2026 at 6:38 AM Weston Pace <[email protected]> > > >>> wrote: > > >>>> > > >>>>> +0.5 > > >>>>> > > >>>>> I guess I'm a bit late to the party since we already have > > >>>>> arrow.parquet.variant but it feels like parquet-specific types > would > > >>> just > > >>>>> fit the definition for community extension types[1]. In other > words, > > >>>>> Parquet is free to define whatever extension types they want and > > there > > >>> is > > >>>>> no need for Arrow PMC approval in the same way that GeoArrow is > free > > to > > >>>>> define whatever extension types they want. > > >>>>> > > >>>>>> To ensure full interoperability, Arrow requires a matching > canonical > > >>>>>> in-memory extension type so that file fields do not lose their > > >>>>>> semantic metadata during serialization across IPC streams, the C > > >>>>>> Data Interface, Arrow Flight, and language bindings. > > >>>>> > > >>>>> I also somewhat disagree with this characterization. As long as > the > > >>>>> producer and consumer agree on the type then there is no need for > > every > > >>>>> middleman (or even Arrow itself) to be aware of it. That's the > > entire > > >>>> goal > > >>>>> of extension types. IPC streams, the C data interface, Arrow > flight, > > >>> and > > >>>>> language bindings should all respect existing metadata and should > > _not_ > > >>>>> strip it out, even if they don't recognize it. > > >>>>> > > >>>>> That being said, I am happy for someone (even if it is Arrow and > not > > >>>>> Parquet) to bless a canonical extension type for any > Parquet-specific > > >>>> types > > >>>>> and so I will turn my vote into a +1 if needed for passage. > > Otherwise > > >>>> I'll > > >>>>> leave it as a half vote to discourage Arrow taking on the mantle of > > >>>>> blessing another format's types. > > >>>>> > > >>>>> [1] > > >>>>> > > >>>>> > > >>>> > > >>> > > > https://arrow.apache.org/docs/format/CanonicalExtensions.html#community-extension-types > > >>>>> > > >>>>> On Fri, Nov 6, 2026 at 11:36 AM Mandukhai Alimaa < > > >>>>> [email protected]> > > >>>>> wrote: > > >>>>> > > >>>>>> Hi everyone, > > >>>>>> > > >>>>>> The FILE column type is being adopted across Parquet, Delta Lake, > > >>>>>> Apache Iceberg, and Apache Spark so the broader ecosystem can > > >>> natively > > >>>>>> leverage structured file-level references. To ensure full > > >>>>>> interoperability, Arrow requires a matching canonical in-memory > > >>>>>> extension type so that file fields do not lose their semantic > > >>> metadata > > >>>>>> during serialization across IPC streams, the C Data Interface, > Arrow > > >>>>>> Flight, and language bindings. > > >>>>>> > > >>>>>> Following up on the discussion thread [1] and the subsequent > format > > >>>>>> change PR [2], we (@rok and I) would like to propose a formal vote > > to > > >>>>>> add arrow.parquet.file as a canonical extension type to the Apache > > >>>>>> Arrow format specification. This proposal already had reviews from > > >>>>>> four different committers on GitHub and closely matches the > Parquet > > >>>>>> file logical type. > > >>>>>> > > >>>>>> [1] > > >>>>> > > https://lists.apache.org/thread.html/5qw2k7rwstk0ol9ymds5qym81myznmpl > > >>>>>> [2] https://github.com/apache/arrow/pull/51248 > > >>>>>> [3] GitHub Issue: https://github.com/apache/arrow/issues/51143 > > >>>>>> > > >>>>>> The vote will be open for at least 72 hours. > > >>>>>> > > >>>>>> Please cast your vote: > > >>>>>> [ ] +1 Accept this proposal > > >>>>>> [ ] +0 > > >>>>>> [ ] -1 Do not accept this proposal because... > > >>>>>> > > >>>>>> Thanks > > >>>>>> > > >>>>> > > >>>> > > >>> > > > > > > > >
