Thanks to everyone for the comments so far. Are there any more comments? I'd
like to start a vote now that we have C++/Java/Go.
Note that after a round of bikeshedding, the proposed type is now called Opaque.
On Fri, Jun 14, 2024, at 15:05, Micah Kornfield wrote:
> I left some comments mostly ar
I left some comments mostly around bike shedding and organization of
description. I agree this is useful.
On Tue, Jun 11, 2024 at 10:22 AM Antoine Pitrou wrote:
>
> Sorry, I had forgotten to comment on this. I think this is generally a
> good idea, but it would obviously need more eyes on it :-
Sorry, I had forgotten to comment on this. I think this is generally a
good idea, but it would obviously need more eyes on it :-)
Can other people go and take a look at David's PR below?
Le 25/05/2024 à 04:47, David Li a écrit :
I've put up a draft PR here: https://github.com/apache/arrow/
gt;> opaque binary (the Postgres COPY representation of the value) but put
>>> >>>>> field metadata so that a consumer can implement a workaround for an
>>> >>>>> unsupported type. It would be arguably better to have implemented
>>>
ntation of the value) but put
>> >>>>> field metadata so that a consumer can implement a workaround for an
>> >>>>> unsupported type. It would be arguably better to have implemented
>> this
>> >>>>> as an extension type; however, field
felt like less of a
commitment when I first worked on this.
Cheers,
-dewey
On Thu, Apr 11, 2024 at 1:20 PM Norman Jordan
wrote:
I was using UUID as an example. It looks like extension types
covers my original request.
From: Felipe Oliveira Carvalho
Sent: Thur
ut
> >>>>> field metadata so that a consumer can implement a workaround for an
> >>>>> unsupported type. It would be arguably better to have implemented
> this
> >>>>> as an extension type; however, field metadata felt like less of a
>
field metadata so that a consumer can implement a workaround for an
>>>>>> unsupported type. It would be arguably better to have implemented this
>>>>>> as an extension type; however, field metadata felt like less of a
>>>>>> commitment when I first worked on this.
>>>>>>
>>
n types covers my
original request.
From: Felipe Oliveira Carvalho
Sent: Thursday, April 11, 2024 7:15 AM
To: dev@arrow.apache.org
Subject: Re: Unsupported/Other Type
The OP used UUID as an example. Would that be enough or the request is for
a flexible mechanism that allows the creatio
t;>> unsupported type. It would be arguably better to have implemented this
> >>>> as an extension type; however, field metadata felt like less of a
> >>>> commitment when I first worked on this.
> >>>>
> >>>> Cheers,
> >>
worked on this.
>>>>
>>>> Cheers,
>>>>
>>>> -dewey
>>>>
>>>> On Thu, Apr 11, 2024 at 1:20 PM Norman Jordan
>>>> wrote:
>>>>>
>>>>> I was using UUID as an example. It looks like exte
n types covers my
original request.
From: Felipe Oliveira Carvalho
Sent: Thursday, April 11, 2024 7:15 AM
To: dev@arrow.apache.org
Subject: Re: Unsupported/Other Type
The OP used UUID as an example. Would that be enough or the request is for
a flexible mechanism that allows the cr
hu, Apr 11, 2024 at 1:20 PM Norman Jordan
>> wrote:
>>>
>>> I was using UUID as an example. It looks like extension types covers my
>>> original request.
>>>
>>> From: Felipe Oliveira Carvalho
>>> Sent: Thursday
s using UUID as an example. It looks like extension types covers my
>> original request.
>>
>> From: Felipe Oliveira Carvalho
>> Sent: Thursday, April 11, 2024 7:15 AM
>> To: dev@arrow.apache.org
>> Subject: Re: Unsupported/Other Type
>>
lipe Oliveira Carvalho
> Sent: Thursday, April 11, 2024 7:15 AM
> To: dev@arrow.apache.org
> Subject: Re: Unsupported/Other Type
>
> The OP used UUID as an example. Would that be enough or the request is for
> a flexible mechanism that allows the creation of one-off nominal typ
I was using UUID as an example. It looks like extension types covers my
original request.
From: Felipe Oliveira Carvalho
Sent: Thursday, April 11, 2024 7:15 AM
To: dev@arrow.apache.org
Subject: Re: Unsupported/Other Type
The OP used UUID as an example. Would
One-off nominal types can already be created as application-specific
extension types.
The specific thing about UUID, JSON and a couple other types is that
they exist in many systems already, so a standardized way of conveying
them with Arrow would enhance interoperation between all these syst
The OP used UUID as an example. Would that be enough or the request is for
a flexible mechanism that allows the creation of one-off nominal types for
very specific use-cases?
—
Felipe
On Thu, 11 Apr 2024 at 05:06 Antoine Pitrou wrote:
>
> Yes, JSON and UUID are obvious candidates for new canoni
Yes, JSON and UUID are obvious candidates for new canonical extension
types. XML also comes to mind, but I'm not sure there's much of a use
case for it.
Regards
Antoine.
Le 10/04/2024 à 22:55, Wes McKinney a écrit :
In the past we have discussed adding a canonical type for UUID and JSON.
wrote:
> It’s worth noting that this maps well to the User data type field in the
> XdbcTypeInfo APIs for Flight SQL.
>
> From: David Li
> Date: Wednesday, April 10, 2024 at 3:23 PM
> To: dev@arrow.apache.org
> Subject: Re: Unsupported/Other Type
> I think this should be a
It’s worth noting that this maps well to the User data type field in the
XdbcTypeInfo APIs for Flight SQL.
From: David Li
Date: Wednesday, April 10, 2024 at 3:23 PM
To: dev@arrow.apache.org
Subject: Re: Unsupported/Other Type
I think this should be an extension type, yes.
It could be
I think this should be an extension type, yes.
It could be parametrized on the storage type; the other system might at least
know that one type is based on another (e.g. a user defined type). Type
metadata can be preserved in the extension type's metadata.
I think it would be good to have stand
In the past we have discussed adding a canonical type for UUID and JSON. I
still think this is a good idea and could improve ergonomics in downstream
language bindings (e.g. by exposing JSON querying function or automatically
boxing UUIDs in built-in UUID types, like the Python uuid library). Has
a
Hi Norman,
Arrow has a concept of extension types [1] along with the possibility of
proposing new canonical extension types [2]. This seems to cover the
use-cases you mention but I might be misunderstanding?
Thanks,
Micah
[1]
https://arrow.apache.org/docs/format/Columnar.html#format-metadata-ext
Problem Description
Currently Arrow schemas can only contain columns of types supported by Arrow.
In some cases an Arrow schema maps to an external schema. This can result in
the Arrow schema not being able to support all the columns from the external
schema.
Consider an external system that c
25 matches
Mail list logo
