TimDaly wrote: > All non-bit floating point output involves a choice of rounding. > You have chosen to do rounding by truncation. This choice is not > obvious nor necessarily the best. See > http://www.diycalculator.com/popup-m-round.shtml > > In any case, you're welcome to test any way you prefer. > I just thought I'd give you insights from Axiom's tests. > I'll let it rest. > > Tim
I must admit, it had crossed my mind that truncation like this was not optimal. Even without reading the reference, I would have thought something like testing ifL abs(expected - got) < some_tolerable_absolute_error may have been a better choice. Truncating is obviously going to be a problem if the expected number is 1.000000000000000 but you get 0.9999999999999999 Those two numbers are fairly close in value, yet their digits are completely different. Perhaps I'm missing something here. Dave -- To post to this group, send an email to sage-devel@googlegroups.com To unsubscribe from this group, send an email to sage-devel+unsubscr...@googlegroups.com For more options, visit this group at http://groups.google.com/group/sage-devel URL: http://www.sagemath.org
