Have you considered using fixed-length binary values for these?
Crypto algorithms might logically be defined in terms of mathematical
operations on integers, but their efficient implementation tends to feature
inlined operations at the machine word level instead of generic add, div,
mod, mul opera
Hi Will,
I'll also note that, while float16 is a first-class datatype, I'm not
sure any Arrow implementation is able to do anything else than just
transport it currently.
You're right that we'd probably want extension number types to be based
on fixed-size-binary. A complication is endiann
A further advantage of third-party extension types is that they give you a
way to experiment without as much concern for compatibility.
I think writing an extension type if possible, and promoting it to an
official type (extension or otherwise) only if necessary, is a good general
approach.
On Tu
Hello Arrow devs,
I actually have a use case where we'd like to support a new number type in
Arrow, but instead of larger numbers, smaller ones. :) For machine learning
use cases, we at Lance would like to support bfloat16 [1]. These are 16-bit
floating point numbers that trade significant digits
Your question seems unspecific, but we now have the possibility of
standardizing canonical extension types (which are, of course, optional
to implement and support):
https://arrow.apache.org/docs/format/CanonicalExtensions.html
Le 23/05/2023 à 19:45, Ian Joiner a écrit :
That’s a possibil
That’s a possibility. Do we consider officially support them?
On Tuesday, May 23, 2023, Antoine Pitrou wrote:
>
> I'm not sure what you're actually proposing here. A new extension type
> perhaps?
>
>
> Le 23/05/2023 à 19:13, Ian Joiner a écrit :
>
>> Hi,
>>
>> We need to have really large integ
I'm not sure what you're actually proposing here. A new extension type
perhaps?
Le 23/05/2023 à 19:13, Ian Joiner a écrit :
Hi,
We need to have really large integers (with 128, 256 and 512 bits) as well
as decimals (up to at least decimal1024) because they do actually exist in
crypto / web
