On 29.12.2015 16:58, Ronald S. Bultje wrote:
> Hi,
>
> On Tue, Dec 29, 2015 at 10:55 AM, Andreas Cadhalpun <
> andreas.cadhal...@googlemail.com> wrote:
>
>>> +static av_always_inline void idct_1D_8(pixel *x, int xstride, const
>> pixel y[16])
>>> +{
>>> + int t1h, t4h, t6h, t0 = y[0], t1 = y[1], t2 = y[2], t3 = y[3], t4 =
>> y[4];
>>> + int t5 = y[5], t6 = y[6], t7 = y[7];
>>> + t5 -= (t3*2485 + 4096) >> 13;
>>> + t3 += (t5*18205 + 16384) >> 15;
>>> + t5 -= (t3*2485 + 4096) >> 13;
>>> + t7 -= (t1*3227 + 16384) >> 15;
>>> + t1 += (t7*6393 + 16384) >> 15;
>>> + t7 -= (t1*3227 + 16384) >> 15;
>>> + t1 += t3;
>>
>> These seven lines can overflow.
>
>
> Why do you believe they can overflow?
Because ubsan told me that.
> Look at range constraints for the input values.
Apparently that is not constrained enough, e.g.:
t3 = -1449866
t3*2485 = -3602917010 < INT32_MIN
Best regards,
Andreas
_______________________________________________
ffmpeg-devel mailing list
ffmpeg-devel@ffmpeg.org
http://ffmpeg.org/mailman/listinfo/ffmpeg-devel
