Livecode can handle floating point numbers up to 2^52. But then calculations would have to fit within that range in a floating point perspective too. So, once you add that ninth zero to the end of your divider, it struggles with the calculation and returns, effectively, number out of range.
Fractions in FP binary are tremendously complicated. Calculation in it are even more so. https://www.h-schmidt.net/FloatConverter/IEEE754.html On Sat, 20 Mar 2021 at 20:23, Paul Dupuis via use-livecode < use-livecode@lists.runrev.com> wrote: > I don't have an answer, other than to say there has been a LOT of > discussion over the years about the limits of number precision in > LiveCode and a number of enhancement requests and 3rd party libraries > that have attempted to provide a framework for high precision math. You > might search the list archives or search the forum on the LC web site > for past discussions about number precision. > > On 3/20/2021 3:06 PM, jbv via use-livecode wrote: > > Hi list, > > I am a bit lost : > > > > set numberformat to ".#########################" > > put -0.0000007347880794884118999 / 1000 > > -- returns -0.0000000007347880794884119 which is ok > > > > put -0.0000007347880794884118999 / 1000000 > > -- returns -0.0000000000007347880794884 -> also ok > > > > put -0.0000007347880794884118999 / 100000000 > > -- returns -0.0000000000000073478807949 -> still ok > > > > put -0.0000007347880794884118999 / 1000000000 > > -- returns 0 -> ????? > > > > Any explanations ? > > Thanks in advance. > > Best, > > jbv > > > > _______________________________________________ > > use-livecode mailing list > > use-livecode@lists.runrev.com > > Please visit this url to subscribe, unsubscribe and manage your > > subscription preferences: > > http://lists.runrev.com/mailman/listinfo/use-livecode > > > _______________________________________________ > use-livecode mailing list > use-livecode@lists.runrev.com > Please visit this url to subscribe, unsubscribe and manage your > subscription preferences: > http://lists.runrev.com/mailman/listinfo/use-livecode > -- Pi Digital _______________________________________________ use-livecode mailing list use-livecode@lists.runrev.com Please visit this url to subscribe, unsubscribe and manage your subscription preferences: http://lists.runrev.com/mailman/listinfo/use-livecode
