Whats wrong with #roundedTo: ?
If you actually need it calculated to 8 significant digits then you can use scaled decimals e.g. (0.09560268 asScaledDecimal - 0.005 asScaledDecimal) see https://0.30000000000000004.com for more about why floating point math does this sometimes Donald Howard wrote > Hello everyone, > > This is an issue I've had awareness of going way back but it's the first > time I've had to personally deal with it. I've done some research and > found it addressed in a number of ways but given current time pressures > and > a huge list of to-dos, I thought I'd turn to the community for rapid > advice > to get me over the hump. The circumstances are summarized below in code > and comments suitable for a playground. > > "This is the actual arithmetic I want to do... > 0.09560268 - 0.005 = 0.09060268" > > "Evaluate this in a playground and you'll see this is what > floating point arithmetic produces --the first two are wrong, every one > after is correct" > 0.09560268 - 0.005 "=> 0.09060267999999999". > 0.0956026 - 0.005 "=> 0.09060259999999999". > 0.095602 - 0.005 "=> 0.090602". > 0.09560 - 0.005 "=> 0.0906". > 0.0956 - 0.005 "=> 0.0906". > 0.095 - 0.005 "=> 0.09". > > "This workaround works" > 0.09560268 - 0.005 roundTo: 0.00000001 "=> 0.09060268". > "but one has to compute number of digits of precision > and according to study above, it's not necessary at > precisions >= 0.000001" > > "What are other suggestions, workarounds or approaches community has?" > "THANKS!" > > > Don Howard > > <https://objectguild.com> > *Founding Member* > dhoward@ > +31653139744 > (US) 651-253-7024 -- Sent from: http://forum.world.st/Pharo-Smalltalk-Users-f1310670.html
