[Bug target/90094] New: better handling of x == LONG_MIN on x86-64

[drepper.fsp+rhbz at gmail dot com]

https://gcc.gnu.org/bugzilla/show_bug.cgi?id=90094

            Bug ID: 90094
           Summary: better handling of x == LONG_MIN on x86-64
           Product: gcc
           Version: 9.0
            Status: UNCONFIRMED
          Severity: normal
          Priority: P3
         Component: target
          Assignee: unassigned at gcc dot gnu.org
          Reporter: drepper.fsp+rhbz at gmail dot com
  Target Milestone: ---

Compile the following on x86-64:

unsigned f(long a)
{
  return a == LONG_MIN;
}

The result for -O3 is:

f:      movabs $0x8000000000000000,%rax
        cmp    %rax,%rdi
        sete   %al
        movzbl %al,%eax
        retq   

With -Os it looks like this:

f:      mov    $0x1,%eax
        shl    $0x3f,%rax
        cmp    %rax,%rdi
        sete   %al
        movzbl %al,%eax
        retq   

I think for both optimization directions the code should be compiled as if for
this:

unsigned f(long a)
{
  long r;
  return __builtin_sub_overflow(a, 1, &r);
}

This compiled to

f:      xor    %eax,%eax
        add    $0xffffffffffffffff,%rdi
        seto   %al
        retq   

This should be faster and is definitely shorter than even the -Os version.

For 32-bit x86 the problem doesn't exist is this form, I think.  But it might
apply to some RISC targets as well.