Paul, Sean,
Thank you both for your answers, although it doesn't really solve my
problem.
The most surprising thing is that javascript runs these calculations
without
problem :
(-0.0000007347880794884118999 / 1000000000) returns
-7.347880794884119e-16
(-0.0000007347880794884118999 / 1000000000000) returns
-7.347880794884119e-19
Best
Le 2021-03-21 07:11, Sean Cole (Pi) via use-livecode a écrit :
Livecode can handle floating point numbers up to 2^52. But then
calculations would have to fit within that range in a floating point
perspective too. So, once you add that ninth zero to the end of your
divider, it struggles with the calculation and returns, effectively,
number
out of range.
Fractions in FP binary are tremendously complicated. Calculation in it
are
even more so.
https://www.h-schmidt.net/FloatConverter/IEEE754.html
On Sat, 20 Mar 2021 at 20:23, Paul Dupuis via use-livecode <
use-livecode@lists.runrev.com> wrote:
I don't have an answer, other than to say there has been a LOT of
discussion over the years about the limits of number precision in
LiveCode and a number of enhancement requests and 3rd party libraries
that have attempted to provide a framework for high precision math.
You
might search the list archives or search the forum on the LC web site
for past discussions about number precision.
On 3/20/2021 3:06 PM, jbv via use-livecode wrote:
> Hi list,
> I am a bit lost :
>
> set numberformat to ".#########################"
> put -0.0000007347880794884118999 / 1000
> -- returns -0.0000000007347880794884119 which is ok
>
> put -0.0000007347880794884118999 / 1000000
> -- returns -0.0000000000007347880794884 -> also ok
>
> put -0.0000007347880794884118999 / 100000000
> -- returns -0.0000000000000073478807949 -> still ok
>
> put -0.0000007347880794884118999 / 1000000000
> -- returns 0 -> ?????
>
> Any explanations ?
> Thanks in advance.
> Best,
> jbv
>
> _______________________________________________
> use-livecode mailing list
> use-livecode@lists.runrev.com
> Please visit this url to subscribe, unsubscribe and manage your
> subscription preferences:
> http://lists.runrev.com/mailman/listinfo/use-livecode
_______________________________________________
use-livecode mailing list
use-livecode@lists.runrev.com
Please visit this url to subscribe, unsubscribe and manage your
subscription preferences:
http://lists.runrev.com/mailman/listinfo/use-livecode
_______________________________________________
use-livecode mailing list
use-livecode@lists.runrev.com
Please visit this url to subscribe, unsubscribe and manage your subscription
preferences:
http://lists.runrev.com/mailman/listinfo/use-livecode
