A client wants to know why his db number -9.85 gets displayed by some simple
code as -9.8
I looked at the number and see that
>>> -9.85
-9.8499999999999996
ie I expect simple rounding to produce the observed result and indeed
>>> '%.1f' % -9.85
'-9.8'
however, when I use round I get an unexpected result ie
>>> round(-9.85,1)
-9.9000000000000004
according to its definition
round(x[, n])ΒΆ
Return the floating point value x rounded to n digits after the decimal
point.
If n is omitted, it defaults to zero. The result is a floating point number.
Values are rounded to the closest multiple of 10 to the power minus n;
if two multiples are equally close, rounding is done away from 0 (so. for
example,
round(0.5) is 1.0 and round(-0.5) is -1.0).
so looking at the absolute differences I see
>>> abs(-9.9 - -9.85)
0.050000000000000711
>>> abs(-9.8 - -9.85)
0.049999999999998934
ie the -9.8 value appears closer and at least to a primitive test
>>> abs(-9.9 - -9.85) > abs(-9.8 - -9.85)
True
the distance from the -9.9 result is larger, however, that may be because the
model numbers for -9.8 & -9.9 differ in distance from the true 10**-n values eg
>>> -9.9
-9.9000000000000004
>>> -9.8
-9.8000000000000007
What value should round(-9.85,1) return? Is the result explainable in python (ie
without resort to the internal FP representations etc etc)?
--
Robin Becker
--
http://mail.python.org/mailman/listinfo/python-list
