Re: [FFmpeg-devel] [PATCH 4/9] simple_idct10: improve precision

2015-10-12 Thread Michael Niedermayer
On Mon, Oct 12, 2015 at 07:37:45PM +0200, Christophe Gisquet wrote: > omse goes from 0.03060703 (which fails for dct-test) to 0.01663750. > This also actually improve the error of decoding the sample generated > by fate-vsynth3-dnxhd1080i-10bit using simple_idct10 to FAANI, which > goes (when resam

[FFmpeg-devel] [PATCH 4/9] simple_idct10: improve precision

2015-10-12 Thread Christophe Gisquet
omse goes from 0.03060703 (which fails for dct-test) to 0.01663750. This also actually improve the error of decoding the sample generated by fate-vsynth3-dnxhd1080i-10bit using simple_idct10 to FAANI, which goes (when resampled to yuv422p) from: stddev:0.06 PSNR: 72.28 MAXDIFF:1 to identica