Yes, this would be for an extension type.
On Wed, Apr 17, 2024, at 23:25, Weston Pace wrote: >> people generally find use in Arrow schemas independently of concrete data. > > This makes sense. I think we do want to encourage use of Arrow as a "type > system" even if there is no data involved. And, given that we cannot > easily change a field's data type property to "optional" it makes sense to > use a dedicated type and I so I would be in favor of such a proposal (we > may eventually add an "unknown type" concept in Substrait as well, it's > come up several times, and so we could use this in that context). > > I think that I would still prefer a canonical extension type (with storage > type null) over a new dedicated type. > > On Wed, Apr 17, 2024 at 5:39 AM Antoine Pitrou <anto...@python.org> wrote: > >> >> Ah! Well, I think this could be an interesting proposal, but someone >> should put a more formal proposal, perhaps as a draft PR. >> >> Regards >> >> Antoine. >> >> >> Le 17/04/2024 à 11:57, David Li a écrit : >> > For an unsupported/other extension type. >> > >> > On Wed, Apr 17, 2024, at 18:32, Antoine Pitrou wrote: >> >> What is "this proposal"? >> >> >> >> >> >> Le 17/04/2024 à 10:38, David Li a écrit : >> >>> Should I take it that this proposal is dead in the water? While we >> could define our own Unknown/Other type for say the ADBC PostgreSQL driver >> it might be useful to have a singular type for consumers to latch on to. >> >>> >> >>> On Fri, Apr 12, 2024, at 07:32, David Li wrote: >> >>>> I think an "Other" extension type is slightly different than an >> >>>> arbitrary extension type, though: the latter may be understood >> >>>> downstream but the former represents a point at which a component >> >>>> explicitly declares it does not know how to handle a field. In this >> >>>> example, the PostgreSQL ADBC driver might be able to provide a >> >>>> representation regardless, but a different driver (or say, the JDBC >> >>>> adapter, which cannot necessarily get a bytestring for an arbitrary >> >>>> JDBC type) may want an Other type to signal that it would fail if >> asked >> >>>> to provide particular columns. >> >>>> >> >>>> On Fri, Apr 12, 2024, at 02:30, Dewey Dunnington wrote: >> >>>>> Depending where your Arrow-encoded data is used, either extension >> >>>>> types or generic field metadata are options. We have this problem in >> >>>>> the ADBC Postgres driver, where we can convert *most* Postgres types >> >>>>> to an Arrow type but there are some others where we can't or don't >> >>>>> know or don't implement a conversion. Currently for these we return >> >>>>> opaque binary (the Postgres COPY representation of the value) but put >> >>>>> field metadata so that a consumer can implement a workaround for an >> >>>>> unsupported type. It would be arguably better to have implemented >> this >> >>>>> as an extension type; however, field metadata felt like less of a >> >>>>> commitment when I first worked on this. >> >>>>> >> >>>>> Cheers, >> >>>>> >> >>>>> -dewey >> >>>>> >> >>>>> On Thu, Apr 11, 2024 at 1:20 PM Norman Jordan >> >>>>> <norman.jor...@improving.com.invalid> wrote: >> >>>>>> >> >>>>>> I was using UUID as an example. It looks like extension types >> covers my original request. >> >>>>>> ________________________________ >> >>>>>> From: Felipe Oliveira Carvalho <felipe...@gmail.com> >> >>>>>> Sent: Thursday, April 11, 2024 7:15 AM >> >>>>>> To: dev@arrow.apache.org <dev@arrow.apache.org> >> >>>>>> Subject: Re: Unsupported/Other Type >> >>>>>> >> >>>>>> The OP used UUID as an example. Would that be enough or the request >> is for >> >>>>>> a flexible mechanism that allows the creation of one-off nominal >> types for >> >>>>>> very specific use-cases? >> >>>>>> >> >>>>>> — >> >>>>>> Felipe >> >>>>>> >> >>>>>> On Thu, 11 Apr 2024 at 05:06 Antoine Pitrou <anto...@python.org> >> wrote: >> >>>>>> >> >>>>>>> >> >>>>>>> Yes, JSON and UUID are obvious candidates for new canonical >> extension >> >>>>>>> types. XML also comes to mind, but I'm not sure there's much of a >> use >> >>>>>>> case for it. >> >>>>>>> >> >>>>>>> Regards >> >>>>>>> >> >>>>>>> Antoine. >> >>>>>>> >> >>>>>>> >> >>>>>>> Le 10/04/2024 à 22:55, Wes McKinney a écrit : >> >>>>>>>> In the past we have discussed adding a canonical type for UUID >> and JSON. >> >>>>>>> I >> >>>>>>>> still think this is a good idea and could improve ergonomics in >> >>>>>>> downstream >> >>>>>>>> language bindings (e.g. by exposing JSON querying function or >> >>>>>>> automatically >> >>>>>>>> boxing UUIDs in built-in UUID types, like the Python uuid >> library). Has >> >>>>>>>> anyone done any work on this to anyone's knowledge? >> >>>>>>>> >> >>>>>>>> On Wed, Apr 10, 2024 at 3:05 PM Micah Kornfield < >> emkornfi...@gmail.com> >> >>>>>>>> wrote: >> >>>>>>>> >> >>>>>>>>> Hi Norman, >> >>>>>>>>> Arrow has a concept of extension types [1] along with the >> possibility of >> >>>>>>>>> proposing new canonical extension types [2]. This seems to >> cover the >> >>>>>>>>> use-cases you mention but I might be misunderstanding? >> >>>>>>>>> >> >>>>>>>>> Thanks, >> >>>>>>>>> Micah >> >>>>>>>>> >> >>>>>>>>> [1] >> >>>>>>>>> >> >>>>>>>>> >> >>>>>>> >> https://arrow.apache.org/docs/format/Columnar.html#format-metadata-extension-types >> >>>>>>>>> [2] >> https://arrow.apache.org/docs/format/CanonicalExtensions.html >> >>>>>>>>> >> >>>>>>>>> On Wed, Apr 10, 2024 at 11:44 AM Norman Jordan >> >>>>>>>>> <norman.jor...@improving.com.invalid> wrote: >> >>>>>>>>> >> >>>>>>>>>> Problem Description >> >>>>>>>>>> >> >>>>>>>>>> Currently Arrow schemas can only contain columns of types >> supported by >> >>>>>>>>>> Arrow. In some cases an Arrow schema maps to an external >> schema. This >> >>>>>>> can >> >>>>>>>>>> result in the Arrow schema not being able to support all the >> columns >> >>>>>>> from >> >>>>>>>>>> the external schema. >> >>>>>>>>>> >> >>>>>>>>>> Consider an external system that contains a column of type >> UUID. To >> >>>>>>> model >> >>>>>>>>>> the schema in Arrow, the user has two choices: >> >>>>>>>>>> >> >>>>>>>>>> 1. Do not include the UUID column in the Arrow schema >> >>>>>>>>>> >> >>>>>>>>>> 2. Map the column to an existing Arrow type. This will >> not include >> >>>>>>> the >> >>>>>>>>>> original type information. A UUID can be mapped to a >> FixedSizeBinary, >> >>>>>>> but >> >>>>>>>>>> consumers of the Arrow schema will be unable to distinguish a >> >>>>>>>>>> FixedSizeBinary field from a UUID field. >> >>>>>>>>>> >> >>>>>>>>>> Possible Solution >> >>>>>>>>>> >> >>>>>>>>>> * Add a new type code that represents unsupported types >> >>>>>>>>>> >> >>>>>>>>>> * Values for the new type are represented as variable >> length >> >>>>>>> binary >> >>>>>>>>>> >> >>>>>>>>>> Some drivers can expose data even when they don’t understand >> the data >> >>>>>>>>>> type. For example, the PostgreSQL driver will return the raw >> bytes for >> >>>>>>>>>> fields of an unknown type. Using an explicit type lets clients >> know >> >>>>>>> that >> >>>>>>>>>> they should convert values if they were able to determine the >> actual >> >>>>>>> data >> >>>>>>>>>> type. >> >>>>>>>>>> >> >>>>>>>>>> Questions >> >>>>>>>>>> >> >>>>>>>>>> * What is the impact on existing clients when they >> encounter >> >>>>>>> fields >> >>>>>>>>> of >> >>>>>>>>>> the unsupported type? >> >>>>>>>>>> >> >>>>>>>>>> * Is it safe to assume that all unsupported values can >> safely be >> >>>>>>>>>> converted to a variable length binary? >> >>>>>>>>>> >> >>>>>>>>>> * How can we preserve information about the original >> type? >> >>>>>>>>>> >> >>>>>>>>>> >> >>>>>>>>> >> >>>>>>>> >> >>>>>>> >> >>>>>> Warning: The sender of this message could not be validated and may >> not be the actual sender. >>
