LC has a lot of "moving part" going on behind the scenes and much of it is
not immediately obvious. I can definitely say this list is a great resource
and between the suggestions, the clarifications, and my somewhat kludgy work
around I have a path forward.
Thanks,
John
Arithmetic takes (for each parameter) either a string that is a numeral or it
takes the result of arithmetic (or sometimes binaryDecode).
The result of arithmetic is a "double precision" floating point number, an
internal representation. That essentially means you get about 15 digits of
resolu
Not sure it will work as you need it too, but one last test before I crash.
I put a calc into a field and did this:
*put* format("%1.60e",value(field 1))
Bipassing do, and it seems to function well enough. Would have to wrap it
in a try/catch or something since you can't count on the user put
Mike,
I need to dig a bit deeper and perhaps make a test app. The issue could be
a bit more complicated in my case as the math is happening behind a "do"
statement.
There is field in which the user enters what they want to calculate. I take
that and add a "put" and an "into" targeting
Hmm. was just messing with it in the message box.. I'm not a math guy, but
seems that when using format you shouldn't need to specify the numberformat.
I did this.. put format("%1.60e",1.23456e-41 * 1e-20) where 60 is the
desired precision. Seems to work, same type of result but you specify the
Craig,
With an insanely large number of "#" in number format I can get this to
work: 1234567890123e-10 * 1e-170. I get the result: 1.2345678901230e-168.
It appears that there is a bit more going on as they are floating point
values.
Thanks,
John
On Jun 23, 2013, at 7:35 PM, Craig Newman
Mike,
Thanks for the tip. I am using "format" for display purposes so that I get
correctly formatted numbers but, unless I use "numberFormat" first, there is
not enough precision available from the result of the multiplication and the
result is 0.
It appears that "numberFormat" sets the
I may be old fashioned, in that I always thought that the Apple SANE limits, 19
digits, was still the law of the land. This is OS9 stuff, though and I could
just be doddering. But it would mean that setting the number format to such an
outrageous string would be futile, most of that string ignor
look at the format function, specifically the "%#.##E" (or e) incantation.
On Sun, Jun 23, 2013 at 8:20 PM, John wrote:
>It seems odd that I haven't stumbled into this before and I am sure I
> am doing something wrong, I just can't seem to find a better way around the
> issue of working wit
It seems odd that I haven't stumbled into this before and I am sure I am
doing something wrong, I just can't seem to find a better way around the issue
of working with small numbers.
Suppose I want to multiply two small numbers such as: 1.23456e-41 * 1e-20.
Unless I am happy with a resul
10 matches
Mail list logo
