Hi Diego, During my investigations on CVE-2015-8216 (I didn't update the tracker because I'm still working on it, by the way), I discovered the following floating point exception in libswscale/utils.c:
$ ./avconv -i fpe.jpg out.jpg
avconv version 0.8.21, Copyright (c) 2000-2014 the Libav developers
built on Dec 12 2017 14:57:35 with gcc 7.2.1 20171205
Input #0, image2, from '../samples/output.jpg':
Duration: 00:00:00.04, start: 0.000000, bitrate: N/A
Stream #0.0: Video: mjpeg, yuvj444p, 658x61951, 25 tbr, 25 tbn, 25 tbc
Incompatible pixel format 'yuvj444p' for codec 'mjpeg', auto-selecting format
'yuvj420p'
[buffer @ 0x60a0000002a0] w:658 h:61951 pixfmt:yuvj444p
[avsink @ 0x60a0000003e0] auto-inserting filter 'auto-inserted scaler 0'
between the filter 'src' and the filter 'out'
[scale @ 0x60a000000520] w:658 h:61951 fmt:yuvj444p -> w:658 h:61951
fmt:yuvj420p flags:0x4
ASAN:DEADLYSIGNAL
=================================================================
==21916==ERROR: AddressSanitizer: FPE on unknown address 0x561e62fa19cc (pc
0x561e62fa19cc bp 0x7fff06ec4300 sp 0x7fff06ec4130 T0)
#0 0x561e62fa19cb in initFilter libswscale/utils.c:504
#1 0x561e62fa6b72 in sws_init_context libswscale/utils.c:986
#2 0x561e62fa91c8 in sws_getContext libswscale/utils.c:1131
#3 0x561e61f8a6f0 in config_props libavfilter/vf_scale.c:215
#4 0x561e61f62cbc in avfilter_config_links libavfilter/avfilter.c:191
#5 0x561e61f67e2c in ff_avfilter_graph_config_links
libavfilter/avfiltergraph.c:120
#6 0x561e61f69074 in avfilter_graph_config libavfilter/avfiltergraph.c:245
#7 0x561e61f23cb3 in configure_video_filters
/home/hle/Development/C/libav/libav-0.8.21/avconv.c:617
#8 0x561e61f38641 in transcode_init
/home/hle/Development/C/libav/libav-0.8.21/avconv.c:2475
#9 0x561e61f3a800 in transcode
/home/hle/Development/C/libav/libav-0.8.21/avconv.c:2680
#10 0x561e61f4c1a7 in main
/home/hle/Development/C/libav/libav-0.8.21/avconv.c:4551
#11 0x7fdf26779560 in __libc_start_main
(/lib/x86_64-linux-gnu/libc.so.6+0x20560)
#12 0x561e61f20ca9 in _start
(/home/hle/Development/C/libav/libav-0.8.21/avconv+0x17eca9)
AddressSanitizer can not provide additional info.
SUMMARY: AddressSanitizer: FPE libswscale/utils.c:504 in initFilter
==21916==ABORTING
libav 0.8.21 crashes, but neither 9.21 nor ffmpeg. As far as I am aware this
issue hasn't been reported until now so it might be worth taking a look
at it.
Program received signal SIGFPE, Arithmetic exception.
0x00005555567539cc in initFilter (outFilter=0x62a000012ad8,
filterPos=0x62a000012af8, outFilterSize=0x62a000012b0c, xInc=131070,
srcW=61951, dstW=30976, filterAlign=2, one=4096, flags=4, cpu_flags=21467,
srcFilter=0x0, dstFilter=0x0,
param=0x62a000012270, is_horizontal=0)
at libswscale/utils.c:504
504 int intV= ROUNDED_DIV(v, sum);
(gdb) l
494 int j;
495 int64_t error=0;
496 int64_t sum=0;
497
498 for (j=0; j<filterSize; j++) {
499 sum+= filter[i*filterSize + j];
500 }
501 sum= (sum + one/2)/ one;
502 for (j=0; j<*outFilterSize; j++) {
503 int64_t v= filter[i*filterSize + j] + error;
504 int intV= ROUNDED_DIV(v, sum);
505 (*outFilter)[i*(*outFilterSize) + j]= intV;
506 error= v - intV*sum;
507 }
508 }
(gdb) p v
$1 = 0
(gdb) p sum
$2 = 0
(gdb) p filterSize
$3 = 8
(gdb) p filter[i*filterSize]
$4 = 0
(gdb) p filter[i*filterSize + 1]
$5 = 0
(gdb) p filter[i*filterSize + 2]
$6 = 0
[... snip]
(gdb) p filter[i*filterSize + 7]
$11 = 0
(gdb) p filter[i*filterSize + 8]
$12 = 0
So obviously the issue comes from the fact that the filter is full
of zeroes at these positions, which leads to an unwanted zero sum
variable.
It looks like the filter starts to be full of zeroes starting at
position 65533 (~ 2^16 = 65536, coincidence ?).
Is there a valid reason for a filter to be full of zeroes ?
Regards,
Hugo
--
Hugo Lefeuvre (hle) | www.owl.eu.com
4096/ 9C4F C8BF A4B0 8FC5 48EB 56B8 1962 765B B9A8 BACA
signature.asc
Description: PGP signature
