"Vince" <vincecar...@gmail.com> writes:
> If I break this down the following returns 1:
> SELECT (cos(radians(37.7438640)) * cos(radians(37.7438640)) *
> cos(radians(-97.4631299) - radians(-97.4631299)) + sin(radians(37.7438640))
> * sin(radians(37.743864000)));
No, it doesn't return 1, it returns 1-plus-a-little-bit, which is hidden
by float8out's desire to not print things like 1.0000000000000002.
Try it after setting extra_float_digits to 2 or so, or do
regression=# SELECT (cos(radians(37.7438640)) * cos(radians(37.7438640)) *
cos(radians(-97.4631299) - radians(-97.4631299)) + sin(radians(37.7438640))
* sin(radians(37.743864000))) - 1;
?column?
----------------------
2.22044604925031e-16
(1 row)
As somebody already remarked, you need to use a version of that formula
that's less prone to roundoff error.
regards, tom lane
--
Sent via pgsql-bugs mailing list (pgsql-bugs@postgresql.org)
To make changes to your subscription:
http://www.postgresql.org/mailpref/pgsql-bugs
