[Bug tree-optimization/100256] New: spurious stringop-overflow warning with memset(..., sizeof(dest)) on variable-length array at -O3

[gandalf at winds dot org via Gcc-bugs]

https://gcc.gnu.org/bugzilla/show_bug.cgi?id=100256

            Bug ID: 100256
           Summary: spurious stringop-overflow warning with memset(...,
                    sizeof(dest)) on variable-length array at -O3
           Product: gcc
           Version: 10.3.0
            Status: UNCONFIRMED
          Severity: normal
          Priority: P3
         Component: tree-optimization
          Assignee: unassigned at gcc dot gnu.org
          Reporter: gandalf at winds dot org
  Target Milestone: ---

When 'j_degree' is unknown per the function below, -O3 causes a
stringop-overflow warning to be emitted on memset() with strange region sizes.
The code snapshot below is the result of trying to simplify/remove as many
lines as I could while still causing the warning to generate.

GCC 10.3.0 and GCC 11.0.1 commit a6f018fcc6ce9236ff37eac33b01a0a80103c9f6,
running on x86_64-pc-linux-gnu (Gentoo):

---

typedef long unsigned int size_t;

extern void *memset (void *__s, int __c, size_t __n) __attribute__
((__nothrow__ , __leaf__)) __attribute__ ((__nonnull__ (1)));

extern void *calloc (size_t __nmemb, size_t __size)
     __attribute__ ((__nothrow__ , __leaf__)) __attribute__ ((__malloc__))
__attribute__ ((__alloc_size__ (1, 2))) ;

static void setup_matrix(double **ppd_xx, double *pd_xy, int j_degree)
{
  int kk;
  double ad_xsum[j_degree*2 + 1];

  memset(ad_xsum,0,sizeof(ad_xsum));

  for(kk=0; kk < j_degree*2 + 1; kk++) {
    ad_xsum[kk]++;
    if(kk < j_degree + 1)
      pd_xy[kk]++;
  }
}

void polyfit(int j_degree, double ad_coef[], double *pd_xy, double **ppd_xx)
{
  int jj;

  for(jj=0;jj<j_degree+1;jj++)
    if(!(ppd_xx[jj] = calloc(j_degree+1,sizeof(double))))
      return;

  setup_matrix(ppd_xx,pd_xy,j_degree);
}

---

gcc-10.3.0 -O3 -c bound.c

In function ‘setup_matrix’,
    inlined from ‘polyfit’ at bound.c:30:3:
bound.c:13:3: warning: ‘memset’ writing between 18446744056529682440 and
18446744073709551608 bytes into a region of size between 18446744056529682440
and 18446744073709551608 [-Wstringop-overflow=]
bound.c: In function ‘polyfit’:
bound.c:11:10: note: at offset 0 to an object with size between
18446744056529682440 and 18446744073709551608 declared here

---

gcc-11.0.1 -O3 -c bound.c

In function ‘setup_matrix’,
    inlined from ‘polyfit’ at bound.c:30:3:
bound.c:13:3: warning: ‘memset’ writing between 18446744056529682440 and
18446744073709551608 bytes into a region of size 9223372036854775807
[-Wstringop-overflow=]
bound.c: In function ‘polyfit’:
bound.c:11:10: note: destination object ‘ad_xsum’ of size 9223372036854775807