Re: Incorrect results from numeric round() and trunc()

2024-07-09 Thread Dean Rasheed
On Mon, 8 Jul 2024 at 11:08, Joel Jacobson wrote: > > LGTM. > Thanks for the review. I have pushed and back-patched this. Regards, Dean

Re: Incorrect results from numeric round() and trunc()

2024-07-08 Thread Joel Jacobson
On Mon, Jul 8, 2024, at 11:45, Dean Rasheed wrote: > On Mon, 8 Jul 2024 at 00:40, Joel Jacobson wrote: >> >> On Sun, Jul 7, 2024, at 13:28, Dean Rasheed wrote: >> > I've also tidied up a bit by replacing all instances of SHRT_MAX with >> > a new constant NUMERIC_WEIGHT_MAX, whose name more accurat

Re: Incorrect results from numeric round() and trunc()

2024-07-08 Thread Dean Rasheed
On Mon, 8 Jul 2024 at 00:40, Joel Jacobson wrote: > > On Sun, Jul 7, 2024, at 13:28, Dean Rasheed wrote: > > I've also tidied up a bit by replacing all instances of SHRT_MAX with > > a new constant NUMERIC_WEIGHT_MAX, whose name more accurately > > describes the limit, as used in various other ove

Re: Incorrect results from numeric round() and trunc()

2024-07-07 Thread Joel Jacobson
On Sun, Jul 7, 2024, at 13:28, Dean Rasheed wrote: > I've also tidied up a bit by replacing all instances of SHRT_MAX with > a new constant NUMERIC_WEIGHT_MAX, whose name more accurately > describes the limit, as used in various other overflow checks. Having thought a bit more on this, I think we

Re: Incorrect results from numeric round() and trunc()

2024-07-07 Thread Joel Jacobson
On Sun, Jul 7, 2024, at 13:28, Dean Rasheed wrote: > The numeric round() and trunc() functions clamp the scale argument to > the range between +/- NUMERIC_MAX_RESULT_SCALE, which is +/- 2000. > That's a long way short of the actual allowed range of type numeric, > so they produce incorrect results

Incorrect results from numeric round() and trunc()

2024-07-07 Thread Dean Rasheed
The numeric round() and trunc() functions clamp the scale argument to the range between +/- NUMERIC_MAX_RESULT_SCALE, which is +/- 2000. That's a long way short of the actual allowed range of type numeric, so they produce incorrect results when rounding/truncating more than 2000 digits before or af