I’m having some trouble comparing numbers with lots of decimal places - numbers that ought to be equal are slightly unequal. In the LC Dictionary entry for ‘numberFormat’ there’s a warning about this, and a suggested solution:
> > Note: Since LiveCode does not use decimal numbers for its internal > calculations (for reasons of speed), the decimal representation of a number > is sometimes slightly off the correct number. For example, 10^-1 is equal to > 0.1, but is calculated (to eighteen decimal places) as 0.100000000000000006. > Because of this, setting the numberFormat to specify many decimal places > after the decimal point may produce unexpected results in a statement that > tests for an exact number. To prevent this, either avoid setting the > numberFormat to a value more precise than you need, or use the abs function > instead of the = operator to test equality: > > set the numberformat to ".##################" > put 10^-1 = 0.1 -- reports false because of the decimal error > put abs((10^-1) - 0.1) = zero -- reports true Well, this sounds good, but looking up the definition of 'abs', there is no suggestion that taking the abs of a number will in any way alter its precision, so what does this advice mean? I have done a test where two numbers differ only after the tenth decimal place, but the test set the numberformat to "0.##########" put abs(aa-bb) = 0 returns false, so it didn't help. Is this just a documentation bug, or is there something funny going on in the LC7.0.2-rc-2? Graham _______________________________________________ use-livecode mailing list use-livecode@lists.runrev.com Please visit this url to subscribe, unsubscribe and manage your subscription preferences: http://lists.runrev.com/mailman/listinfo/use-livecode
