Re: [FFmpeg-devel] [PATCH 20/21] avcodec/smacker: Use unsigned for prediction values

2020-08-19 Thread Paul B Mahol
On 8/1/20, Andreas Rheinhardt wrote: > Up until now, the Smacker decoder has pretended that the prediction > values are signed in code like 'pred[0] += (unsigned)sign_extend(val, 16)' > (the cast has been added to this code later to fix undefined behaviour). > This has been even done in case the P

[FFmpeg-devel] [PATCH 20/21] avcodec/smacker: Use unsigned for prediction values

2020-08-01 Thread Andreas Rheinhardt
Up until now, the Smacker decoder has pretended that the prediction values are signed in code like 'pred[0] += (unsigned)sign_extend(val, 16)' (the cast has been added to this code later to fix undefined behaviour). This has been even done in case the PCM format is u8. Yet in case of 8/16 bit samp