On Tue, Jul 2, 2024, at 11:05, Joel Jacobson wrote:
> On Tue, Jul 2, 2024, at 10:22, Dean Rasheed wrote:
>> Shortly after posting that, I realised that there was a small bug. This bit:
>>
>> case 2:
>> newdig = (int) var1digits[1] * var2digits[res_ndigits - 4];
>>
>> isn't quite right in the case where rscale is less than the full
>> result. In that case, the least significant result digit has a
>> contribution from one more var2 digit, so it needs to be:
>>
>> newdig = (int) var1digits[1] * var2digits[res_ndigits - 4];
>> if (res_ndigits - 3 < var2ndigits)
>> newdig += (int) var1digits[0] * var2digits[res_ndigits -
>> 3];
>>
Just to be able to verify mul_var() is working as expected when
rscale is less than the full result, I've added a numeric_mul_patched()
function that takes a third rscale_adjustment parameter:
I've tried to get a different result with and without the fix,
but I'm failing to hit the bug.
Can you think of an example that should trigger the bug?
Example usage:
SELECT numeric_mul_patched(9999.999,2,0);
var1ndigits=1 var2ndigits=2 rscale=3
numeric_mul_patched
---------------------
19999.998
(1 row)
SELECT numeric_mul_patched(9999.999,2,1);
var1ndigits=1 var2ndigits=2 rscale=4
numeric_mul_patched
---------------------
19999.9980
(1 row)
SELECT numeric_mul_patched(9999.999,2,-1);
var1ndigits=1 var2ndigits=2 rscale=2
numeric_mul_patched
---------------------
20000.00
(1 row)
SELECT numeric_mul_patched(9999.999,2,-2);
var1ndigits=1 var2ndigits=2 rscale=1
numeric_mul_patched
---------------------
20000.0
(1 row)
/Joel
