Why not try the round function to limit the number of digits being compared? Put round(myNum,nDigits) - round(num2,nDigits)=theDiff Bill
William Prothero http://es.earthednet.org > On Feb 4, 2015, at 4:17 AM, Peter W A Wood <peterwaw...@gmail.com> wrote: > > Graham > > I think that it is a documentation error. As far as I know, LiveCode uses > binary floating point numbers when performing arithmetic. Not all non-whole > decimal numbers cannot be accurate represented in binary floating point > numbers. I cannot see how just comparing the absolute value of the difference > between two numbers will be any more accurate than comparing the numbers. > > This problem isn’t by any means unique to LiveCode. Here is a Windows > JavaScript example that shows the same behaviour: > > var a = 1.0 + 2.0; > var b = 3.0; > if (a === b) { > res = 'correct'; > }{ > res = 'incorrect'; > } > WScript.Echo('a === b ' + res); > if (Math.abs(a - b) === 0) { > res = 'correct'; > }{ > res = 'incorrect'; > } > WScript.Echo('abs (a - b) === 0 ' + res); > > It displays incorrect for both comparisons. (To run the script save it in a > file with the extension .js and double-click on it - for Windows only) > > The normal workaround in JavaScript is to perform all calculations using > integers and then scaling the numbers down when you want to display them. > > Regards > > Peter > > This is an enhancement request (Bug 9349) for LiveCode to use decimal > floating point numbers in place of binary floating point numbers. > > >> On 4 Feb 2015, at 19:19, Graham Samuel <livf...@mac.com> wrote: >> >> I’m having some trouble comparing numbers with lots of decimal places - >> numbers that ought to be equal are slightly unequal. In the LC Dictionary >> entry for ‘numberFormat’ there’s a warning about this, and a suggested >> solution: >> >>> >>> Note: Since LiveCode does not use decimal numbers for its internal >>> calculations (for reasons of speed), the decimal representation of a number >>> is sometimes slightly off the correct number. For example, 10^-1 is equal >>> to 0.1, but is calculated (to eighteen decimal places) as >>> 0.100000000000000006. Because of this, setting the numberFormat to specify >>> many decimal places after the decimal point may produce unexpected results >>> in a statement that tests for an exact number. To prevent this, either >>> avoid setting the numberFormat to a value more precise than you need, or >>> use the abs function instead of the = operator to test equality: >>> >>> set the numberformat to ".##################" >>> put 10^-1 = 0.1 -- reports false because of the decimal error >>> put abs((10^-1) - 0.1) = zero -- reports true >> >> Well, this sounds good, but looking up the definition of 'abs', there is no >> suggestion that taking the abs of a number will in any way alter its >> precision, so what does this advice mean? I have done a test where two >> numbers differ only after the tenth decimal place, but the test >> >> set the numberformat to "0.##########" >> put abs(aa-bb) = 0 >> >> returns false, so it didn't help. >> >> Is this just a documentation bug, or is there something funny going on in >> the LC7.0.2-rc-2? >> >> Graham >> >> >> >> >> >> _______________________________________________ >> use-livecode mailing list >> use-livecode@lists.runrev.com >> Please visit this url to subscribe, unsubscribe and manage your subscription >> preferences: >> http://lists.runrev.com/mailman/listinfo/use-livecode > > _______________________________________________ > use-livecode mailing list > use-livecode@lists.runrev.com > Please visit this url to subscribe, unsubscribe and manage your subscription > preferences: > http://lists.runrev.com/mailman/listinfo/use-livecode _______________________________________________ use-livecode mailing list use-livecode@lists.runrev.com Please visit this url to subscribe, unsubscribe and manage your subscription preferences: http://lists.runrev.com/mailman/listinfo/use-livecode
