https://gcc.gnu.org/bugzilla/show_bug.cgi?id=96040
--- Comment #2 from Joseph C. Sible <josephcsible at gmail dot com> ---
Andrew Gierth posted this to the Lua mailing list:
> I think I see what's happening here, but I don't think I have an account
> on the gcc bug tracker to post it there (feel free to forward this).
> It's not (I think) a confusion over how many arguments the specialized
> tostringbuff.part.0.isra function has, but rather over the type, or
> equivalently the register allocation, for the first parameter.
>
> The callsite is putting num->value_.n into %rdi, and &space into %rsi,
> as if the function were declared as taking (long long v, char *buf)
> rather than (double v, char *buf) which would require that num->value_.n
> be placed in %xmm0 and &space into %rdi.
>
> But the code of the function itself is assuming that the incoming values
> are in %xmm0 and %rdi - %xmm0 is not touched because it's already in the
> right place for the snprintf call, while %rdi is left as the first arg
> to snprintf. The value 0x3ff0000000000000 is of course 1.0 as a double,
> which naturally does not work well as a pointer so it blows up.
I agree with the gist of that. I think this is more of a calling convention
issue than an argument count issue. It's as if the following pseudo-C were
written:
static int tostringbuff.part.0.isra.0 (union { long long i; double n; } value_
/* rdi */, char *str /* rsi */);
void addnum2buff (int *buff, struct TValue *num) {
if(num->tt_ == 3) {
buff += snprintf(space,50,"%lld",num->value_.i);
} else {
buff += tostringbuff.part.0.isra.0(num->value_ /* rdi */, space /* rsi
*/);
}
}
static int tostringbuff.part.0.isra.0 (double n /* xmm0 */, char *str /* rdi
*/) {
int len;
len = snprintf(str,50,"%.14g",n);
if(str[strspn(str, "-0123456789")] == '\0') {
str[len++] = '.';
str[len++] = '0';
}
return len;
}
In English, the compiler couldn't make up its mind as to how
tostringbuff.part.0.isra.0 was declared. At the call site, the first parameter
was a union, but at the definition, the first parameter was a double. Mixing up
a union and its element is fine when they only live in memory since they have
the same address, but here it was in a register. Since the calling convention
puts those types in different kinds of registers, what the function thought was
a pointer was actually the double value, predictably causing a crash when it
was dereferenced.
