On 12/01/2020 21:13, Donald Howard wrote:
"Evaluate this in a playground and you'll see this is what
floating point arithmetic produces --the first two are wrong, every
one after is correct"
0.09560268 - 0.005 "=> 0.09060267999999999".
0.0956026 - 0.005 "=> 0.09060259999999999".
0.095602 - 0.005 "=> 0.090602".
Wrong in what sense? I am rather sure that the computation is done
correctly, meaning that it follows the rules defined by IEEE standard
754. I am also relatively sure (meaning a bit less) that the conversion
of the binary result to a decimal output string is correct, in the sense
that it applies today's preferred practice of ensuring round-trip
equality. So you probably mean that the result is no what you expected.
The main point to be clear about is if you care about the binary value
or about its decimal representation as a string. If you need the
internal value to be exact, you have to switch to something other than
binary floats - scaled decimals might work for you. If you only want
nice-looking outputs, you can probably get what you want using roundTo:
applied to the result.
Cheers,
Konrad.
