Re: [FFmpeg-devel] [PATCH 1/8] cbs_h264: Fix overflow in shifts

2018-03-13 Thread Michael Niedermayer
On Sun, Mar 11, 2018 at 06:30:14PM +, Mark Thompson wrote: > The type of the result of a shift operation is unaffected by the type of > the right operand, so some existing code overflows with undefined behaviour > when the element length is 32. Add a helper macro to calculate the maximum > val

[FFmpeg-devel] [PATCH 1/8] cbs_h264: Fix overflow in shifts

2018-03-11 Thread Mark Thompson
The type of the result of a shift operation is unaffected by the type of the right operand, so some existing code overflows with undefined behaviour when the element length is 32. Add a helper macro to calculate the maximum value correctly and then use it everywhere this pattern appears. Found-by