[PATCH] Prevent extract_muldiv from introducing an overflow (PR sanitizer/80800)

[Marek Polacek]

extract_muldiv folds 

  (n * 10000 * z) * 50

to

  (n * 500000) * z

which is a wrong transformation to do, because it may introduce an overflow.
This resulted in a ubsan false positive.  So we should just disable this
folding altogether.  Does the approach I took make sense?

Bootstrapped/regtested on x86_64-linux, ok for trunk?

2017-05-19  Marek Polacek  <pola...@redhat.com>

        PR sanitizer/80800
        * fold-const.c (extract_muldiv_1): Don't fold ((X * C1) * Y) * C
        to (X * C2) * Y.

        * c-c++-common/ubsan/pr80800.c: New test.
        * c-c++-common/Wduplicated-branches-1.c: Adjust an expression.

diff --git gcc/fold-const.c gcc/fold-const.c
index 19aa722..e525c2d 100644
--- gcc/fold-const.c
+++ gcc/fold-const.c
@@ -6260,6 +6260,17 @@ extract_muldiv_1 (tree t, tree c, enum tree_code code, 
tree wide_type,
       break;
 
     case MULT_EXPR:
+      /* ((X * C1) * Y) * C
+        cannot be reduced to
+        (X * C2) * Y (where C2 == C * C1)
+        because that can introduce an overflow.  */
+      if (same_p
+         && op0 != NULL_TREE
+         && TREE_CODE (op0) == MULT_EXPR
+         && TREE_CODE (TREE_OPERAND (op0, 1)) == INTEGER_CST
+         && TYPE_OVERFLOW_UNDEFINED (TREE_TYPE (t)))
+       break;
+
       /* We have a special case here if we are doing something like
         (C * 8) % 4 since we know that's zero.  */
       if ((code == TRUNC_MOD_EXPR || code == CEIL_MOD_EXPR
diff --git gcc/testsuite/c-c++-common/Wduplicated-branches-1.c 
gcc/testsuite/c-c++-common/Wduplicated-branches-1.c
index c0b93fc..7c5062d 100644
--- gcc/testsuite/c-c++-common/Wduplicated-branches-1.c
+++ gcc/testsuite/c-c++-common/Wduplicated-branches-1.c
@@ -89,7 +89,7 @@ f (int i, int *p)
   if (i == 8) /* { dg-warning "this condition has identical branches" } */
     return i * 8 * i * 8;
   else
-    return 8 * i * 8 * i;
+    return i * 8 * i * 8;
 
 
   if (i == 9) /* { dg-warning "this condition has identical branches" } */
diff --git gcc/testsuite/c-c++-common/ubsan/pr80800.c 
gcc/testsuite/c-c++-common/ubsan/pr80800.c
index e69de29..992c136 100644
--- gcc/testsuite/c-c++-common/ubsan/pr80800.c
+++ gcc/testsuite/c-c++-common/ubsan/pr80800.c
@@ -0,0 +1,25 @@
+/* PR sanitizer/80800 */
+/* { dg-do run } */
+/* { dg-options "-fsanitize=undefined -fsanitize-undefined-trap-on-error" } */
+
+int n = 20000;
+int z = 0;
+
+int
+fn1 (void)
+{
+  return (n * 10000 * z) * 50;
+}
+
+int
+fn2 (void)
+{
+  return (10000 * n * z) * 50;
+}
+
+int
+main ()
+{
+  fn1 ();
+  fn2 ();
+}

        Marek