I definitely have the understanding that this is (in some sense) intended as a sort of "mostly transport-only" type, and while I originally thought that this was a fine idea I now have second thoughts. Maybe it's the framing as "Canonical BigDecimal Extension Type"? But given that most of the changes to Arrow in the last two years have come from a desire to improve computational speed, it feels wrong to pick a layout for a "canonical BigDecimal type" without trying to get at least some expert feedback on performance considerations. For instance, wouldn't it be better for the magnitude to be aligned at 4 bytes instead of 3? And what are pros vs cons on having a separate sign bit vs a signed magnitude?
It's a shame that there aren't any existing documented formats that would meet all these requirements. The closest thing I was able to find that's an implementation at least sort of in this space is https://github.com/cockroachdb/apd. It's not quite an in-memory layout, but if you constrain the platform to 64-bits it seems pretty close. On Mon, Aug 24, 2026 at 5:46 PM David Li <[email protected]> wrote: > > I think part of it is that the scope has expanded though the comments, but > perhaps we weren't clear initially: the main motivation is that when > interfacing with RDBMSes, we simply don't have a good way to represent > NUMERIC-like types, because their domain tends to exceed Arrow's decimal > types (higher precision and/or INF/NAN values). Hence the main concerns were > being able to represent these values, being able to stringify for display, > and being able to (fallibly) convert to whatever the "native" BigDecimal type > is; but not necessarily do compute on the values directly. Right now, ADBC > drivers generally stringify the values (sometimes also wrapping in > arrow.opaque), and we were trying to have something a bit more consistent > than that. > > It seems that designing a "proper" BigDecimal type for Arrow involves a lot > of tradeoffs, and frankly I don't want to tackle that unless there is a > present use case for a BigDecimal with compute kernels? > > On Fri, Aug 21, 2026, at 03:27, Micah Kornfield wrote: > > I took another pass through the document. My main concern is how we > > arrived at the chosen representation, it doesn't seem like we are clearly > > articulating additional requirements and using those to drive a decision. > > > > Curt could you maybe raise yours on the doc as well so we can close on them? > > > > Thanks, > > Micah > > > > On Sun, Aug 16, 2026 at 7:35 PM Micah Kornfield <[email protected]> > > wrote: > > > >> Yeah, I think the vote might be slightly premature. I think there were a > >> lot of open questions on representation last time I reviewed the doc. If > >> those have been resolved, having a little more time for review would be > >> worthwhile. > >> > >> -Micah > >> > >> On Sun, Aug 16, 2026 at 10:17 AM Curt Hagenlocher <[email protected]> > >> wrote: > >> > >>> -0 (binding) > >>> > >>> I'd be happy for others to tell me I'm wrong, but I don't see enough > >>> consensus in the (stalled?) discussion on the spec to feel like this > >>> is ready for a vote. I also have other reservations, but I don't want > >>> to hijack a voting thread into a discussion thread, so I'll think > >>> about what I want to say and either leave comments on the spec or > >>> start a new thread. > >>> > >>> > >>> -Curt > >>> > >>> On Thu, Aug 13, 2026 at 1:39 PM Mandukhai Alimaa > >>> <[email protected]> wrote: > >>> > > >>> > Hello everyone, > >>> > > >>> > Following up on the previous discussion thread [1], I would like to > >>> > formally open a VOTE for adopting the BigDecimal canonical extension > >>> > type into the Apache Arrow specification. > >>> > > >>> > Summary: > >>> > The proposed `arrow.big_decimal` canonical extension type provides > >>> > high-fidelity representation and transport for variable-scale > >>> > numeric/decimal data (e.g., PostgreSQL NUMERIC, Trino DECIMAL, Oracle > >>> > NUMBER) without forcing a uniform scale across an entire column. > >>> > > >>> > Format PR: > >>> > Golang implementation draft: > >>> https://github.com/apache/arrow-go/pull/1201 > >>> > Rust implementation draft: > >>> https://github.com/apache/arrow-rs/pull/10669 > >>> > > >>> > Resources: > >>> > Specification proposal: > >>> > > >>> https://docs.google.com/document/d/10YNDPW9068RMsvAV_pjmC-uuerwWEwdhGqJARyVvnWI/edit?tab=t.0#heading=h.6pwu28gkulfs > >>> > > >>> > The vote will be open for at least 72 hours (3 business days). > >>> > > >>> > Please cast your vote: > >>> > [ ] +1 Accept this proposal > >>> > [ ] +0 > >>> > [ ] -1 Do not accept this proposal because... > >>> > > >>> > > >>> > Thank you! > >>> > Mandy > >>> > > >>> > [1] https://lists.apache.org/thread/34opw343xrzkb0vhhbqto9vx061c7j7f > >>> > >>
