Hello, Should we initiate a discussion on the Parquet mailing list?
Regards Zehua Zou > On Nov 9, 2026, at 16:22, Antoine Pitrou <[email protected]> wrote: > > > Hello, > > I have roughly the same position as Weston here. > > Regards > > Antoine. > > > > Le 09/11/2026 à 06:04, Weston Pace a écrit : >>> Are you saying there should be super-project canonical extension types? :) >> No. I am suggesting the Parquet PMC publish an extension-types.md document >> in [1] that describes what the extension type definitions are. This would >> be exactly like [2]. This lets the Parquet team bless whatever >> unambiguous definition they like. Anyone wanting to consume custom Parquet >> types must use those definitions. The Parquet team is then in complete >> control and doesn't need to seek the Arrow team's blessing. The Arrow team >> can then create a link back to that document in [3] in much the same way we >> do for GeoArrow. >> For example, the Substrait type system does not name all Arrow types. in >> Arrow, we publish [4] which describes Substrait extension types to map to >> the existing Arrow types. >> Or, as another example, Lance's blob type is an Arrow extension type >> defined in [5]. Consumers of Lance custom types consult the Lance spec for >> those types' definitions. >>> Similarily, I was thinking of adding more canonical Parquet extension >>> types like GEOMETRY and GEOGRAPHY because I found it really tricky >>> when implementing Iceberg V3 types in iceberg-cpp where GeoArrow type >>> were not an option. >> If Iceberg has its own set of types (separate from Parquet and GeoArrow) >> then Iceberg can also maintain an extension-types.md document right? >> [1] https://github.com/apache/parquet-format/ >> [2] https://github.com/geoarrow/geoarrow/blob/main/extension-types.md >> [3] >> https://arrow.apache.org/docs/format/CanonicalExtensions.html#community-extension-types >> [4] >> https://github.com/apache/arrow/blob/main/format/substrait/extension_types.yaml >> [5] >> https://github.com/lance-format/lance/blob/main/docs/src/format/table/blob.md#table-schema >> On Sat, Nov 7, 2026 at 7:00 AM Rok Mihevc <[email protected]> wrote: >>> Thanks for the discussion Weston and Gang! >>> >>>> I also somewhat disagree with this characterization. As long as the >>>> producer and consumer agree on the type then there is no need for every >>>> middleman (or even Arrow itself) to be aware of it. That's the entire >>> goal >>>> of extension types. IPC streams, the C data interface, Arrow flight, and >>>> language bindings should all respect existing metadata and should _not_ >>>> strip it out, even if they don't recognize it. >>> >>> This is a great point and it's ideally how things should work. Now how do >>> we get there? Should Parquet readers that output Arrow map Parquet logical >>> types into "parquet.<logical_type>" extension types? If this requires >>> non-trivial logic, to make the mapping unambiguous, are we implicitly >>> specifying extension types at the conversion layer? >>> >>>> and so I will turn my vote into a +1 if needed for passage. Otherwise >>> I'll >>>> leave it as a half vote to discourage Arrow taking on the mantle of >>>> blessing another format's types. >>> >>> Are you saying there should be super-project canonical extension types? :) >>> >>> >>> Best, >>> Rok >>> >>> On Sat, Nov 7, 2026 at 2:57 AM Gang Wu <[email protected]> wrote: >>> >>>> Thanks Weston for raising this! I have a different opinion. >>>> >>>> Canonical extension type for Parquet types like VARIANT、FILE and future >>>> VECTOR brings better interoperability to different Parquet bindings in >>>> Arrow. AFAIK, it would be a headache if different bindings define their >>>> own custom extensions then we end up with different interpretations of >>>> ARROW:schema serialization across bindings. >>>> >>>> Similarily, I was thinking of adding more canonical Parquet extension >>>> types like GEOMETRY and GEOGRAPHY because I found it really tricky >>>> when implementing Iceberg V3 types in iceberg-cpp where GeoArrow type >>>> were not an option. >>>> >>>> On Sat, Nov 7, 2026 at 6:38 AM Weston Pace <[email protected]> >>> wrote: >>>> >>>>> +0.5 >>>>> >>>>> I guess I'm a bit late to the party since we already have >>>>> arrow.parquet.variant but it feels like parquet-specific types would >>> just >>>>> fit the definition for community extension types[1]. In other words, >>>>> Parquet is free to define whatever extension types they want and there >>> is >>>>> no need for Arrow PMC approval in the same way that GeoArrow is free to >>>>> define whatever extension types they want. >>>>> >>>>>> To ensure full interoperability, Arrow requires a matching canonical >>>>>> in-memory extension type so that file fields do not lose their >>>>>> semantic metadata during serialization across IPC streams, the C >>>>>> Data Interface, Arrow Flight, and language bindings. >>>>> >>>>> I also somewhat disagree with this characterization. As long as the >>>>> producer and consumer agree on the type then there is no need for every >>>>> middleman (or even Arrow itself) to be aware of it. That's the entire >>>> goal >>>>> of extension types. IPC streams, the C data interface, Arrow flight, >>> and >>>>> language bindings should all respect existing metadata and should _not_ >>>>> strip it out, even if they don't recognize it. >>>>> >>>>> That being said, I am happy for someone (even if it is Arrow and not >>>>> Parquet) to bless a canonical extension type for any Parquet-specific >>>> types >>>>> and so I will turn my vote into a +1 if needed for passage. Otherwise >>>> I'll >>>>> leave it as a half vote to discourage Arrow taking on the mantle of >>>>> blessing another format's types. >>>>> >>>>> [1] >>>>> >>>>> >>>> >>> https://arrow.apache.org/docs/format/CanonicalExtensions.html#community-extension-types >>>>> >>>>> On Fri, Nov 6, 2026 at 11:36 AM Mandukhai Alimaa < >>>>> [email protected]> >>>>> wrote: >>>>> >>>>>> Hi everyone, >>>>>> >>>>>> The FILE column type is being adopted across Parquet, Delta Lake, >>>>>> Apache Iceberg, and Apache Spark so the broader ecosystem can >>> natively >>>>>> leverage structured file-level references. To ensure full >>>>>> interoperability, Arrow requires a matching canonical in-memory >>>>>> extension type so that file fields do not lose their semantic >>> metadata >>>>>> during serialization across IPC streams, the C Data Interface, Arrow >>>>>> Flight, and language bindings. >>>>>> >>>>>> Following up on the discussion thread [1] and the subsequent format >>>>>> change PR [2], we (@rok and I) would like to propose a formal vote to >>>>>> add arrow.parquet.file as a canonical extension type to the Apache >>>>>> Arrow format specification. This proposal already had reviews from >>>>>> four different committers on GitHub and closely matches the Parquet >>>>>> file logical type. >>>>>> >>>>>> [1] >>>>> https://lists.apache.org/thread.html/5qw2k7rwstk0ol9ymds5qym81myznmpl >>>>>> [2] https://github.com/apache/arrow/pull/51248 >>>>>> [3] GitHub Issue: https://github.com/apache/arrow/issues/51143 >>>>>> >>>>>> The vote will be open for at least 72 hours. >>>>>> >>>>>> Please cast your vote: >>>>>> [ ] +1 Accept this proposal >>>>>> [ ] +0 >>>>>> [ ] -1 Do not accept this proposal because... >>>>>> >>>>>> Thanks >>>>>> >>>>> >>>> >>> >
