Hi,
after enabling the altivec optimizations on ppc64el, many fate tests
failed, some of them with 'stack smashing detected'. For the full build
log see [1].
The attached patch fixes all the stack smashing crashes.
But I don't know, what causes the other test failures.
Rong Yan, maybe you can have a look?
Best regards,
Andreas
1:
https://buildd.debian.org/status/fetch.php?pkg=ffmpeg&arch=ppc64el&ver=7%3A2.5.3-1&stamp=1421583916
>From 93d465397d95594870b2ff9bbd662ccfa306bfb7 Mon Sep 17 00:00:00 2001
From: Andreas Cadhalpun <andreas.cadhal...@googlemail.com>
Date: Tue, 20 Jan 2015 18:46:01 +0100
Subject: [PATCH] libavcodec/ppc/mpegvideoencdsp.c: fix stack smashing in
pix_norm1_altivec() and pix_sum_altivec()
The vec_ste calls were mistakenly changed to vec_vsx_st in c5ca76a, which
caused stack smashing.
Changing them back fixes crashes on ppc64el, when configured with
--toolchain=hardened.
---
libavcodec/ppc/mpegvideoencdsp.c | 4 ++--
1 file changed, 2 insertions(+), 2 deletions(-)
diff --git a/libavcodec/ppc/mpegvideoencdsp.c b/libavcodec/ppc/mpegvideoencdsp.c
index e91ba5d..3e6765c 100644
--- a/libavcodec/ppc/mpegvideoencdsp.c
+++ b/libavcodec/ppc/mpegvideoencdsp.c
@@ -55,7 +55,7 @@ static int pix_norm1_altivec(uint8_t *pix, int line_size)
/* Sum up the four partial sums, and put the result into s. */
sum = vec_sums((vector signed int) sv, (vector signed int) zero);
sum = vec_splat(sum, 3);
- vec_vsx_st(sum, 0, &s);
+ vec_ste(sum, 0, &s);
return s;
}
#else
@@ -113,7 +113,7 @@ static int pix_sum_altivec(uint8_t *pix, int line_size)
/* Sum up the four partial sums, and put the result into s. */
sumdiffs = vec_sums((vector signed int) sad, (vector signed int) zero);
sumdiffs = vec_splat(sumdiffs, 3);
- vec_vsx_st(sumdiffs, 0, &s);
+ vec_ste(sumdiffs, 0, &s);
return s;
}
#else
--
2.1.4
_______________________________________________
ffmpeg-devel mailing list
ffmpeg-devel@ffmpeg.org
http://ffmpeg.org/mailman/listinfo/ffmpeg-devel
