[Bug c++/70126] New: VLA accepted in sizeof and typedef, allowing integer overflow

[msebor at gcc dot gnu.org]

https://gcc.gnu.org/bugzilla/show_bug.cgi?id=70126

            Bug ID: 70126
           Summary: VLA accepted in sizeof and typedef, allowing integer
                    overflow
           Product: gcc
           Version: 6.0
            Status: UNCONFIRMED
          Severity: normal
          Priority: P3
         Component: c++
          Assignee: unassigned at gcc dot gnu.org
          Reporter: msebor at gcc dot gnu.org
  Target Milestone: ---

G++ 4.9.3 added support for variable-length arrays specified in WG21 document
N3639 (http://www.open-std.org/jtc1/sc22/wg21/docs/papers/2013/n3639.html). 
That document specifies, among other restrictions, that:

* The sizeof operator shall not be applied to [...] an array of runtime
bound...
* A typedef-name shall not name an array of runtime bound.

G++ 4.9 fails to enforce these restrictions, making it possible for the
definition of a VLA type to cause an integer overflow.

The following test case (which is invalid, according to N3639) shows the
problems.

$ cat v.c && /home/msebor/build/gcc-4.9.3/gcc/xg++
-B/home/msebor/build/gcc-4.9.3/gcc -Wall -Wextra -L
/home/msebor/build/gcc-4.9.3/x86_64-unknown-linux-gnu/libstdc++-v3/src/.libs
-std=c++11 -xc++ v.c && ./a.out 
typedef __SIZE_TYPE__ size_t;

void __attribute__ ((noclone, noinline)) bar (size_t m)
{
    typedef int A [m];
    typedef A A2 [4];

    __builtin_printf ("sizeof (A)  = %zu\nsizeof (A2) = %zu\n",
                     sizeof (A), sizeof (A2));

    if (sizeof (A2) < sizeof (A))
        __builtin_abort ();
}

int main ()
{
    try {
        bar (__SIZE_MAX__ / sizeof (int));
        __builtin_trap ();
    }
    catch (...) {
        __builtin_printf ("exception caught\n");
    }
}
sizeof (A)  = 18446744073709551612
sizeof (A2) = 18446744073709551600
Aborted (core dumped)