[PATCH] Fix qsort ordering violation in tree-vrp.c

[Yuri Gribov]

Hi,

This is a follow-up on https://gcc.gnu.org/ml/gcc/2017-07/msg00078.html

compare_assert_loc in tree-vrp.c could return unpredictable results
due to implicit conversion of unsigned subtraction to int here:
  return ha - hb;

This could return inconsistent results for objects with the following hashes:
  a: 0xdc8e4f72U
  b: 0x912ab538U
  c: 0x5ae66e3bU
Then
  a > b because (int)(0xdc8e4f72U - 0x912ab538U) > 0
  b > c because (int)(0x912ab538U - 0x5ae66e3bU) > 0
but
  a < c because (int)(0xdc8e4f72U - 0x5ae66e3bU) == (int)0x81a7e137U < 0

Bug was found with https://github.com/yugr/sortcheck

Bootstrapped and regtested in x64, ok for trunk?

-Y

fix-order-1.patch
Description: Binary data