[PATCH] tree-optimization/98758 - fix integer arithmetic in data-ref analysis

[Richard Biener]

This fixes some int arithmetic issues and a bogus truncation.

Bootstrapped and tested on x86_64-unknown-linux-gnu, pushed.

2021-01-20  Richard Biener  <rguent...@suse.de>

        PR tree-optimization/98758
        * tree-data-ref.c (int_divides_p): Use lambda_int arguments.
        (lambda_matrix_right_hermite): Avoid undefinedness with
        signed integer abs and multiplication.
        (analyze_subscript_affine_affine): Use lambda_int.

        * gcc.dg/torture/pr98758.c: New testcase.
---
 gcc/testsuite/gcc.dg/torture/pr98758.c | 13 +++++++++++++
 gcc/tree-data-ref.c                    | 12 ++++++------
 2 files changed, 19 insertions(+), 6 deletions(-)
 create mode 100644 gcc/testsuite/gcc.dg/torture/pr98758.c

diff --git a/gcc/testsuite/gcc.dg/torture/pr98758.c 
b/gcc/testsuite/gcc.dg/torture/pr98758.c
new file mode 100644
index 00000000000..7b9fdb2b613
--- /dev/null
+++ b/gcc/testsuite/gcc.dg/torture/pr98758.c
@@ -0,0 +1,13 @@
+/* { dg-do compile } */
+
+long *a, *b;
+long c;
+void d(void)
+{
+  b = a;
+  while (c) {
+    *a = (__INTPTR_TYPE__)(a += (long)1 << (sizeof(long) * 8 - 10));
+    c = b[0];
+    b = a;
+  }
+}
diff --git a/gcc/tree-data-ref.c b/gcc/tree-data-ref.c
index 65fe6d5da91..9d07b415e9d 100644
--- a/gcc/tree-data-ref.c
+++ b/gcc/tree-data-ref.c
@@ -143,7 +143,7 @@ tree_fold_divides_p (const_tree a, const_tree b)
 /* Returns true iff A divides B.  */
 
 static inline bool
-int_divides_p (int a, int b)
+int_divides_p (lambda_int a, lambda_int b)
 {
   return ((b % a) == 0);
 }
@@ -4280,10 +4280,10 @@ lambda_matrix_right_hermite (lambda_matrix A, int m, 
int n,
 
                  a = S[i-1][j];
                  b = S[i][j];
-                 sigma = (a * b < 0) ? -1: 1;
-                 a = abs_hwi (a);
-                 b = abs_hwi (b);
-                 factor = sigma * (a / b);
+                 sigma = ((a < 0) ^ (b < 0)) ? -1: 1;
+                 unsigned HOST_WIDE_INT abs_a = absu_hwi (a);
+                 unsigned HOST_WIDE_INT abs_b = absu_hwi (b);
+                 factor = sigma * (lambda_int)(abs_a / abs_b);
 
                  lambda_matrix_row_add (S, n, i, i-1, -factor);
                  std::swap (S[i], S[i-1]);
@@ -4309,7 +4309,7 @@ analyze_subscript_affine_affine (tree chrec_a,
                                 tree *last_conflicts)
 {
   unsigned nb_vars_a, nb_vars_b, dim;
-  HOST_WIDE_INT gamma, gcd_alpha_beta;
+  lambda_int gamma, gcd_alpha_beta;
   lambda_matrix A, U, S;
   struct obstack scratch_obstack;
 
-- 
2.26.2