[Bug c/44736] New: Overeager -O1 optimization results in incorrect code generation

[opensource3141 at gmail dot com]

I have found what appears to be a pretty serious issue with the new
optimization algorithms in GCC 4.5.0.  The following code (with appropriate
comments inline) demonstrate the problem.  Based on a quick search, I don't
think this bug has been reported yet.  Note that I have assigned the component
to C since I wasn't sure which other component -- rtl-optimization or
tree-optimization or some other component entirely -- was the best component.

I'm fairly certain that the problem is target independent since I see it with
GCC 4.5.0 for mingw-w64 targets.  When compiling with my system GCC (version
4.4.3) or a self-compiled GCC 4.4.4, there are no problems.

Please let me know if you need any other information from me.

---
#include <stdlib.h>
#include <stdio.h>
#include <string.h>
#include <malloc.h>

void *malloc_hook (size_t, const void *);
void  free_hook   (void *, const void *);

void *(* _old_malloc_hook) (size_t, const void *);
void  (* _old_free_hook)   (void *, const void *);

static int _allocCount;

int main (int argc, char *argv[])
{
    _allocCount    = 0;

    // Save any existing malloc hooks
    _old_malloc_hook = __malloc_hook;
    _old_free_hook   = __free_hook;

    // Add our function as the primary __malloc_hook
    __malloc_hook    = malloc_hook;
    __free_hook      = free_hook;

    // Invoke malloc, which should invoke our function
    char *str = (char *)malloc(4);

    strcpy(str, "foo");
    printf("str = %s\n", str);
    printf("allocCount = %d\n", _allocCount);

    return 0;
}

void *malloc_hook (size_t size, const void *caller)
{
    void *result;

    // Unhook ourselves before calling malloc,
    // or else there will be an infinite recursion!
    //
    // *** NOTE THIS IS THE CODE THAT GETS ERRONEOUSLY
    // OPTIMIZED OUT RESULTING IN STACK OVERFLOW. ***
    __malloc_hook = _old_malloc_hook;
    __free_hook   = _old_free_hook;

    // Since malloc is a function and the optimizer has
    // no idea what it does with __malloc_hook, it has
    // to ensure that the above code is generated, or
    // else there will be big problems.
    result = malloc(size);

    _allocCount++;

    // Save the current malloc hooks in case
    // they changed as a result of calling malloc
    _old_malloc_hook = __malloc_hook;
    _old_free_hook   = __free_hook;

    // Add ourselves back as the hook
    __malloc_hook = malloc_hook;
    __free_hook   = free_hook;

    return result;
}

void free_hook (void *ptr, const void *caller)
{
    // Same as above but for free()
    __malloc_hook = _old_malloc_hook;
    __free_hook   = _old_free_hook;

    free(ptr);

    _allocCount--;

    _old_malloc_hook = __malloc_hook;
    _old_free_hook   = __free_hook;

    __malloc_hook = malloc_hook;
    __free_hook   = free_hook;
}
---

] x86_64-linux-gcc -g -O1 -o malloc_hook malloc_hook.c && malloc_hook
Segmentation fault (core dumped)


Here is the configuration options for GCC:
---
Configured with: ../../gcc-4.5.0/configure
--prefix=/usr/local/gcc/x86_64-linux-gcc
--with-sysroot=/usr/local/gcc/x86_64-linux-gcc/x86_64-linux/sdk
--program-prefix=x86_64-linux- --enable-languages=c,c++ --build=i386-linux
--host=i386-linux --target=x86_64-linux --enable-multilib
--enable-targets=x86_64-linux,i386-linux --disable-nls --enable-shared
--without-x
---

Note that the behavior also is exhibited when GCC is invoked with --sysroot=/
(basically using the system libraries instead of the sdk).


-- 
           Summary: Overeager -O1 optimization results in incorrect code
                    generation
           Product: gcc
           Version: 4.5.0
            Status: UNCONFIRMED
          Severity: critical
          Priority: P3
         Component: c
        AssignedTo: unassigned at gcc dot gnu dot org
        ReportedBy: opensource3141 at gmail dot com
 GCC build triplet: i386-linux
  GCC host triplet: i386-linux
GCC target triplet: x86_64-linux


http://gcc.gnu.org/bugzilla/show_bug.cgi?id=44736