Libcudf / cuDF have supported 32-bit and 64-bit decimals for a few releases now (as well as 128-bit decimals in the past couple of releases) and they've generally been received positively from the community. Being able to roundtrip them through Arrow would definitely be nice as well!
On Thu, Mar 3, 2022 at 12:06 PM Micah Kornfield <emkornfi...@gmail.com> wrote: > I think this makes sense to add these. Typically when adding new types, > we've waited on the official vote until there are two reference > implementations demonstrating compatibility. > > On Thu, Mar 3, 2022 at 6:55 AM Antoine Pitrou <anto...@python.org> wrote: > > > > > Hello, > > > > Currently, the Arrow format specification restricts the bitwidth of > > decimal numbers to either 128 or 256 bits. > > > > However, there is interest in allowing other bitwidths, at least 32 and > > 64 bits for this proposal. A 64-bit (respectively 32-bit) decimal > > datatype would allow for precisions of up to 18 digits (respectively 9 > > digits), which are sufficient for some applications which are mainly > > looking for exact computations rather than sheer precision. Obviously, > > smaller datatypes are cheaper to store in memory and cheaper to run > > computations on. > > > > For example, the Spark documentation mentions that some decimal types > > may fit in a Java int (32 bits) or long (64 bits): > > > > > https://spark.apache.org/docs/latest/api/java/org/apache/spark/sql/types/DecimalType.html > > > > ... and a draft PR had even been filed for initial support in the C++ > > implementation (https://github.com/apache/arrow/pull/8578). > > > > I am therefore proposing that we relax the wording in the Arrow format > > specification to also allow 32- and 64-bit decimal types. > > > > This is a preliminary discussion to gather opinions and potential > > counter-arguments against this proposal. If no strong counter-argument > > emerges, we will probably run a vote in a week or two. > > > > Best regards > > > > Antoine. > > >
