On Jun 3, 2011, at 11:35 PM, Mark Wieder wrote: > Van- > > Friday, June 3, 2011, 9:38:47 PM, you wrote: > >> Many thanks to Mike and Mark for helping me get my array and >> datagrid issues resolved to date. > >> The latest issue is in calculations. I need to have calculations >> done in floating numbers and then the array finishes calculating it >> truncates at the 6th decimal. I need to go out to at least E-23. >> Is there a way of setting this behaviour somewhere? > > Check out the numberformat property. But that's quite an array you've > got there - I doubt you'll be getting that kind of precision out of > the numeric library. Untested, though - ymmv.
LC uses IEEE double-precision binary floating point numbers for calculations. These are 64 bits: 52 for mantissa, 11 for exponent and 1 for sign. So exponent range won't be a problem, neither will 6 decimal places, as long as the number of significant digits is less than 15. Be sure to round to that least digit, because calcs are in binary. --Jerry Jensen _______________________________________________ use-livecode mailing list use-livecode@lists.runrev.com Please visit this url to subscribe, unsubscribe and manage your subscription preferences: http://lists.runrev.com/mailman/listinfo/use-livecode
