> On 13 Jan 2020, at 00:28, Paul DeBruicker <pdebr...@gmail.com> wrote:
>
> Whats wrong with #roundedTo: ?
>
>
> If you actually need it calculated to 8 significant digits then you can use
> scaled decimals e.g.
>
> (0.09560268 asScaledDecimal - 0.005 asScaledDecimal)
The weird thing is though, that is still does not behave 100% as one would
expect (i.e. it does not seem exact).
0.09560268 asScaledDecimal - 0.005 asScaledDecimal. "0.09060268s8"
0.09560268 asScaledDecimal - 0.005 asScaledDecimal = 0.09060268
asScaledDecimal. "false"
0.09560268 asScaledDecimal - 0.005 asScaledDecimal = 0.09060268s8. "false"
The internal fractions are not the same ...
While
(9560268 / 1e8) - (5 / 1e3) = (9060268 / 1e8). "true"
So, is the equality test of ScaledDecimal correct ?
> see https://0.30000000000000004.com for more about why floating point math
> does this sometimes
>
>
>
>
>
> Donald Howard wrote
>> Hello everyone,
>>
>> This is an issue I've had awareness of going way back but it's the first
>> time I've had to personally deal with it. I've done some research and
>> found it addressed in a number of ways but given current time pressures
>> and
>> a huge list of to-dos, I thought I'd turn to the community for rapid
>> advice
>> to get me over the hump. The circumstances are summarized below in code
>> and comments suitable for a playground.
>>
>> "This is the actual arithmetic I want to do...
>> 0.09560268 - 0.005 = 0.09060268"
>>
>> "Evaluate this in a playground and you'll see this is what
>> floating point arithmetic produces --the first two are wrong, every one
>> after is correct"
>> 0.09560268 - 0.005 "=> 0.09060267999999999".
>> 0.0956026 - 0.005 "=> 0.09060259999999999".
>> 0.095602 - 0.005 "=> 0.090602".
>> 0.09560 - 0.005 "=> 0.0906".
>> 0.0956 - 0.005 "=> 0.0906".
>> 0.095 - 0.005 "=> 0.09".
>>
>> "This workaround works"
>> 0.09560268 - 0.005 roundTo: 0.00000001 "=> 0.09060268".
>> "but one has to compute number of digits of precision
>> and according to study above, it's not necessary at
>> precisions >= 0.000001"
>>
>> "What are other suggestions, workarounds or approaches community has?"
>> "THANKS!"
>>
>>
>> Don Howard
>>
>> <https://objectguild.com>
>> *Founding Member*
>
>> dhoward@
>
>> +31653139744
>> (US) 651-253-7024
>
>
>
>
>
> --
> Sent from: http://forum.world.st/Pharo-Smalltalk-Users-f1310670.html
