[Bug rtl-optimization/49034] New: ARM optimizer generating incorrect code (causing bad pointer dereference)

[michaelatnavman at gmail dot com]

http://gcc.gnu.org/bugzilla/show_bug.cgi?id=49034

           Summary: ARM optimizer generating incorrect code (causing bad
                    pointer dereference)
           Product: gcc
           Version: 4.4.0
            Status: UNCONFIRMED
          Severity: normal
          Priority: P3
         Component: rtl-optimization
        AssignedTo: unassig...@gcc.gnu.org
        ReportedBy: michaelatnav...@gmail.com
              Host: darwin-x86
            Target: arm-eabi
             Build: 4.4.0 (android NDK r5b)


The ARM optimizer appears to sometimes forget that a register has been
overwritten. This especially causes a problem when it uses an overwritten
register to do a memory access.

The general C++ code structure is:
  struct TPointInt
  {
    int x;
    int y;

    TPointInt (int iX, int iY)
    : x(iX), y(iY) { }

    TPointInt operator-(const TPointInt &rhs) const
    {
      return TPointInt(x - rhs.x, y - rhs.y);
    }
  };

  class LineEndPoints
  {
    public:
      TPointInt p0;
      TPointInt p1;
  };

  class Line
  {
    public:
      LineEndPoints mEndPoints;
      int mLength;
  };

  void bar(Line *const lineBuffer, int &nLinesCnt)
  {
    // ...
    Line &curLine = lineBuffer[nLinesCnt];
    const TPointInt lineVec = curLine.mEndPoints.p1 -
                              curLine.mEndPoints.p0; // <<<<
    // ...
  }

  void foo(void)
  {
    static Line lineBuffer[40];
    int nLinesCnt = 0;
    bar(lineBuffer);
  }

The line indicated with "<<<<" essentially does:
  lineVec.x = curLine.mEndPoints.p1.x - curLine.mEndPoints.p0.x;
  lineVec.y = curLine.mEndPoints.p1.y - curLine.mEndPoints.p0.y;

The assembler output is:
  ; 0x83a0083a is a branch target from a single location. It does not fall
  ; through from 0x83a00838. Prior to the branch, r0 is set to contain the
  ; address of curLine.
  0x83a0083a <bar+702>:  ldr     r2, [r0, #8]   ; Get curLine.mEndPoints.p1.x
  0x83a0083c <bar+704>:  ldr     r0, [r0, #0]   ; Get curLine.mEndPoints.p0.x
  0x83a0083e <bar+706>:  subs    r2, r2, r0     ; Subtract
  0x83a00840 <bar+708>:  str     r2, [sp, #44]  ; Store the x result
  0x83a00842 <bar+710>:  ldr     r2, [r0, #4]   ; Get curLine.mEndPoints.p1.y
                                                ; except r0 has been
overwritten
                                                ; above!
  0x83a00844 <bar+712>:  ldr     r1, [r0, #12]  ; Get curLine.mEndPoints.p0.y
  0x83a00846 <bar+714>:  subs    r1, r1, r2     ; Subtract
  0x83a00848 <bar+716>:  movs    r2, #250       ; (from following code,
                                                ; generating the constant 1000)
  0x83a0084a <bar+718>:  lsls    r2, r2, #2     ; (from following code
                                                ; generating the constant 1000)
  0x83a0084c <bar+720>:  subs    r3, r2, r3     ; (from following code, doing
                                                ; 1000 - x)
  0x83a0084e <bar+722>:  str     r1, [sp, #80]  ; Store the y result

It ends up accessing memory at the location specified by the integer
curLine.mEndPoints.p0.x which is unlikely to be valid.

Compiler is g++ 4.4.0 as provided with the Android NDK r5b. Optimization is
turned on (O2). With optimization off, the issue does not occur.

Getting a small, self-contained, example is difficult since it is highly
sensitive to the surrounding code. Filing mainly to allow people using the NDK
to see why they might be having bad pointer accesses in situations where such
issues should be impossible.