On Sun, Jul 18, 2010 at 2:32 PM, Aaron Lewis <aaron.lewis1...@gmail.com> wrote: > -----BEGIN PGP SIGNED MESSAGE----- > Hash: SHA1 > > On 07/18/2010 08:04 PM, Floor Terra wrote: >> On Sun, Jul 18, 2010 at 1:07 PM, Aaron Lewis <aaron.lewis1...@gmail.com> wrote: >>>> What do you mean by "precision"? >>> >>> Kinda of significance digit. For example: >>> B A number `12.340' , and if say it has 2 digits' precision , then we >>> consider the `0' is not accurate , while `.34' is accurate. >> >> What?! > > See: > > http://en.wikipedia.org/wiki/Arithmetic_precision
Yes. 12.340 has five significant digits. If you want to store it with just two significant digits you get 12. > Come on , just for fun , someone asks me if i could print out the > maximum number that can be stored in a double type , and the *precision* > of a double type. The maximum number you can store in a IEEE double is +infinity of course. A 64 bit double has 52-bit mantissa, together with 1 sign bit it gives you almost 16 decimal significant digits. But like other things in life: It's not the size that maters, it's how you use it. (see math(3) and look for ULP) > > And the second , well i tried to divide 2 by 3 , see when i reaches 7 ( > should be 0.66....7 ) , > may not a clever way. Be careful converting your number to decimal if you want to keep maximum precision. -- Floor Terra <flo...@gmail.com> www: http://brobding.mine.nu/
