On Mon, Nov 16, 2015 at 6:24 PM, Ganesh Ajjanagadde <gajja...@mit.edu> wrote: > On Mon, Nov 16, 2015 at 9:48 AM, Daniel Serpell <dserp...@gmail.com> wrote: >> Hi!, >> >> El Fri, Nov 13, 2015 at 11:42:30AM -0500, Ganesh Ajjanagadde escribio: >>> Whoever wrote this stuff had a pretty bad libm - digits differ pretty >>> quickly. >> >> They where correctly rounded to 24bit precision. I don't know if that >> was intentional, so I can't comment on the correctness of the patch. > > Unless I am off here, doing things at 24 bits makes very little sense: > float itself offers 32 bits
Actually floats offer 23-bits of precision for the significand (with 8 for mantissa and one sign bit) - 7-8 digits, so any more precision than that couldn't be stored accurately anymore, so when using single precision like this code, more precision is not needed. Maybe it was even specifically wanted to have values that are represented accurately, but I do not know the history of this code either. - Hendrik _______________________________________________ ffmpeg-devel mailing list ffmpeg-devel@ffmpeg.org http://ffmpeg.org/mailman/listinfo/ffmpeg-devel
