[Bug target/79170] New: memcmp builtin expansion sequence can overflow

[acsawdey at gcc dot gnu.org]

https://gcc.gnu.org/bugzilla/show_bug.cgi?id=79170

            Bug ID: 79170
           Summary: memcmp builtin expansion sequence can overflow
           Product: gcc
           Version: 7.0
            Status: UNCONFIRMED
          Severity: normal
          Priority: P3
         Component: target
          Assignee: acsawdey at gcc dot gnu.org
          Reporter: acsawdey at gcc dot gnu.org
                CC: segher at gcc dot gnu.org, wschmidt at gcc dot gnu.org
  Target Milestone: ---
            Target: powerpc64*-*-*

The sequence generated for memcmp() builtin expansion can overflow in the subf
and produce the wrong result.

#include <stdio.h>
#include <string.h>
#include <stdlib.h>
#include <limits.h>

#define SIZE 16
int main ()
{
  unsigned char buffer1[SIZE] = { 0,0,0,0,0,0,0,1 };
  unsigned char buffer2[SIZE] = { 0x80,0,0,0,0,0,0,3 };

  asm(" ");

  int n =  memcmp(buffer1,buffer2, SIZE);
  printf("%d\n", n);
}

produces 

        ldbrx 9,0,6
        ldbrx 4,0,8
        subf. 4,4,9
        bne 0,.L2
        addi 10,1,120
        ldbrx 9,0,10
        addi 10,1,104
        ldbrx 4,0,10
        subf 4,4,9
.L2:
        cntlzd 4,4
        addis 3,2,.LC0@toc@ha
        addi 3,3,.LC0@toc@l
        addi 4,4,-1
        xori 4,4,0x3f
        bl printf

If the subf result overflows such that the sign bit is not set in r4, then the
wrong result is produced.