On Mon, Sep 17, 2018 at 09:04:06AM +0200, Martin Vignali wrote:
> > also, have you tried adding a small constant to tmp ?
> > i would expect that this or a similar operation would allow moving
> > away from all "unstable" points without really changing the output in a
> > relevant way.
> >
> >
> Can't find an op, in mult mode, in order to not raise the assert
> But if i use add calc instead of mult,
> I can have a bigger tolerance without raising the assert
>
> In 8 bits mode
> Tolerance
> 0.0000000000001 : pass
> 0.000000000001 : fail for i = 1
> 0.00000000001 : fail for i in [1, 2, 4, 8]
> 0.0000000001 : fail for i in [1, 2, 4, 8, 16, 32, 64]
>
> In 16 bits mode
> 0.0000000000001 : pass
> 0.000000000001 : fail for i in [1, 257, 259, 261, 263, 265, 267, 269, 271,
> 273, 275, 277, 279, 281, 65407]
>
> using this code :
> #define assert_stable_float(x) av_assert0((float)(x+0.0000000000001) ==
> (float)(x-0.0000000000001))
> static void inline fill_uint_to_float_lut(SwsContext *c, int bitdepth) {
> static const double float_mult8 = 1.0 / 255.0;
> static const double float_mult16 = 1.0 / 65535.0;
> int i;
> double tmp = 0.;
>
> if (bitdepth == 8) { /*! fill uint8 to float lut */
> for (i = 0; i < 256; ++i){
> c->uint2float_lut[i] = (float)tmp;
> tmp += float_mult8;
> assert_stable_float(tmp);
> }
> } else if (bitdepth == 16) { /*! fill uint16 to float lut */
> for (i = 0; i < 65536; ++i){
> c->uint2float_lut[i] = (float)tmp;
> tmp += float_mult16;
> assert_stable_float(tmp);This is very unstable, alot worse than before, also the C compiler optimizer can replace it by a float_mult16*(i+1) The reason why this works is probably because it skips the 0 element and not for any reason of this being "better" (the table is just shifted by 1 element) 0 will always fail the assert, i missed this before. 0 does not need to be tested, 0 will stay 0 normally in practice probably also there are 2 divisions in this that you can trivially eliminate /255 and /65535 (extra precission beyond IEEE float/double could change these) also the whole could be done with fewer floats and no extra complexity for example: int64_t tmp2 = 16843009LL * i; (float)((double)tmp2 / (1LL<<32)) and int64_t tmp2 = 4295032833LL * i or uint64_t 281479271743489 [...] -- Michael GnuPG fingerprint: 9FF2128B147EF6730BADF133611EC787040B0FAB If you think the mosad wants you dead since a long time then you are either wrong or dead since a long time.
signature.asc
Description: PGP signature
_______________________________________________ ffmpeg-devel mailing list ffmpeg-devel@ffmpeg.org http://ffmpeg.org/mailman/listinfo/ffmpeg-devel
