Re: [DISCUSS] Support of higher bit-width Decimal type

2020-09-26 Thread Wes McKinney
hi Micah -- this makes sense to me On Fri, Sep 25, 2020 at 10:56 PM Micah Kornfield wrote: > > The decimal256 branch now contains sufficient implementations in Java and C++ > to pass round trip integration tests. Some of python interop is missing (but > actively being worked on). > > I'll plan

Re: [DISCUSS] Support of higher bit-width Decimal type

2020-09-25 Thread Micah Kornfield
The decimal256 branch now contains sufficient implementations in Java and C++ to pass round trip integration tests. Some of python interop is missing (but actively being worked on). I'll plan on creating PR to update the specification with a corresponding vote over the next couple of days. One t

Re: [DISCUSS] Support of higher bit-width Decimal type

2020-08-15 Thread Wes McKinney
On Fri, Aug 14, 2020 at 11:17 PM Micah Kornfield wrote: > > Hi Jacques, > > Do we have a good definition of what is necessary to add a new data type? > > Adding a type but not pulling it through most of the code seems less than > > ideal since it means one part of Arrow doesn't work with another (

Re: [DISCUSS] Support of higher bit-width Decimal type

2020-08-14 Thread Micah Kornfield
Hi Jacques, Do we have a good definition of what is necessary to add a new data type? > Adding a type but not pulling it through most of the code seems less than > ideal since it means one part of Arrow doesn't work with another (providing > a less optimal end-user experience). I think what I pro

Re: [DISCUSS] Support of higher bit-width Decimal type

2020-08-14 Thread Jacques Nadeau
Do we have a good definition of what is necessary to add a new data type? Adding a type but not pulling it through most of the code seems less than ideal since it means one part of Arrow doesn't work with another (providing a less optimal end-user experience). For example, would this work include

Re: [DISCUSS] Support of higher bit-width Decimal type

2020-08-05 Thread Micah Kornfield
> > Sounds fine to me. I guess one question is what needs to be formalized > in the Schema.fbs files or elsewhere in the columnar format > documentation (and we will need to hold an associated vote for that I > think) Yes, i think we will need to hold a vote for it. Since this is essentially a "n

Re: [DISCUSS] Support of higher bit-width Decimal type

2020-08-05 Thread Wes McKinney
Sounds fine to me. I guess one question is what needs to be formalized in the Schema.fbs files or elsewhere in the columnar format documentation (and we will need to hold an associated vote for that I think) On Mon, Aug 3, 2020 at 11:30 PM Micah Kornfield wrote: > > Given no objections, we'll go

Re: [DISCUSS] Support of higher bit-width Decimal type

2020-08-03 Thread Micah Kornfield
Given no objections, we'll go ahead and start implementing support for 256-bit decimals. I'm considering setting up another branch to develop all the components so they can be merged to master atomically. Thanks, Micah On Tue, Jul 28, 2020 at 6:39 AM Wes McKinney wrote: > Generally this sounds

Re: [DISCUSS] Support of higher bit-width Decimal type

2020-07-28 Thread Wes McKinney
Generally this sounds fine to me. At some point it would be good to add 32-bit and 64-bit decimal support but this can be done in the future. On Tue, Jul 28, 2020 at 7:28 AM Fan Liya wrote: > > Hi Micah, > > Thanks for opening the discussion. > I am aware of some scenarios where decimal requires

Re: [DISCUSS] Support of higher bit-width Decimal type

2020-07-28 Thread Fan Liya
Hi Micah, Thanks for opening the discussion. I am aware of some scenarios where decimal requires more than 16 bytes, so I think it would be beneficial to support this in Arrow. Best, Liya Fan On Tue, Jul 28, 2020 at 11:12 AM Micah Kornfield wrote: > Hi Arrow Dev, > ZetaSQL (Google's open sour

[DISCUSS] Support of higher bit-width Decimal type

2020-07-27 Thread Micah Kornfield
Hi Arrow Dev, ZetaSQL (Google's open source standard SQL library) recently introduced a BigNumeric [1] type which requires a 256 bit width to properly support it. I'd like to add support (possibly in collaboration with some of my colleagues) to add support for 256 bit width Decimals in Arrow to sup