On 7/7/21 3:16 PM, Morten Welinder wrote:
> Gnumeric's rounding here is indeed done with
>
> round_to_int(x * 10^d) / 10^d
>
> except that round_to_int deliberately misrounds 0.5-1ulp to 1.
I observe that it's not quite that simple.
I assume the intent is the following:
If the input differs from the nearest integer by only one epsilon,
gnumeric uses the integer value instead.
Call this scheme [A].
Let's focus attention on the int() spreadsheet function.
For positive numbers, I observe that the behavior can be
described as follows:
apply scheme [A], then call POSIX floor().
In contrast, however, for negative numbers I observe that
scheme [A] is not successfully applied. The observed behavior
is identical to POSIX floor().
Diagrams of this are here:
https://www.av8n.com/hack/img48/non-posix-floor_1.png
https://www.av8n.com/hack/img48/non-posix-floor_-1.png
The spreadsheet used to compute the diagrams is here:
https://www.av8n.com/hack/non-posix-floor.gnumeric
I am aware that the function go_fake_floor() in file
...../goffice/goffice/math/go-math.c
contains code that seemingly tries to apply scheme [A],
but even so, the observed numerical result does not conform
to scheme [A].
I don't much care, because I try not to do calculations that
are sensitive to the 16th decimal place. Furthermore, it's
easy to defend against the depredations of scheme [A].
Even so, the observed behavior sure is weird.
_______________________________________________
gnumeric-list mailing list
gnumeric-list@gnome.org
https://mail.gnome.org/mailman/listinfo/gnumeric-list
