Line 72 of libswscale/input.c is: dstU[i] = (ru*r + gu*g + bu*b + (0x10001<<(RGB2YUV_SHIFT-1))) >> RGB2YUV_SHIFT;
The definition of macro RGB2YUV_SHIFT in libswscale/swscale_internal.h is on line 417: #define RGB2YUV_SHIFT 15 By examining the result of executing line 72 in input.c it appears that the radix used for macro RGB2YUV_SHIFT is hexadecimal. So that 0x10001 is shifted left by 20 in subexpression 0x10001<<(RGB2YUV_SHIFT-1) with the result being 0x100000. I was expecting a left-shift of 14, given the macro definition. Should the macro be interpreted as decimal 15 or hexadecimal 15? Thanks. Dan. _______________________________________________ ffmpeg-devel mailing list ffmpeg-devel@ffmpeg.org http://ffmpeg.org/mailman/listinfo/ffmpeg-devel
