Hello gcc and libtool lists, Summary: both Autoconf-generated configure tests as well as some Libtool construct invoke undefined behavior. Question is how to deal with it, and whether GCC, as QoI, may want to define behavior in these cases.
1) Autoconf-generated configure tests often fake the prototype of some function; e.g., AC_CHECK_FUNC([func]) uses char func(); and tries to link that. Using this is undefined according to C99, if func has a different actual prototype, and when all system libraries are LTO'ed, gcc -flto may even detect this kind of inconsistency and could act accordingly (nasal demons and such). 2) libtool has a feature that makes it extract symbol lists from objects and turn them into fake declarations and function/object pointers: fake static preloaded modules. It currently works by running nm or a similar tool over the object, then converting the output with a couple of sed script or so (global_symbol_pipe, global_symbol_to_cdecl, and a couple more) to a synthesized extra source file that then contains code like this: extern int func(); extern char variable; typedef struct { const char *name; void *address; } lt_dlsymlist; extern const lt_dlsymlist lt__PROGRAM__LTX_preloaded_symbols[]; const lt_dlsymlist lt__PROGRAM__LTX_preloaded_symbols[] = { { "@PROGRAM@", (void *) 0 }, {"func", (void *) &func}, {"variable", (void *) &variable}, {0, (void *) 0} }; This is invoking undefined behavior in a couple of respects: a) Pointers to functions are stored in pointer-to-void variables. This could be fixed with an incompatible API change to using a union of object and function pointer, I guess. b) The symbols 'func' and 'variable' likely have the wrong prototypes, i.e., elsewhere, they might be declared as void func(int, double); double variable[42]; instead. I haven't come across any actual issues with this yet, except now LTO may rightfully complain about it. Question is, what can we do about this? We could ensure to never pass -flto or -fwhopr to the compilation of the libtool symfile object, and remove it from some or all link tests done in configure. That's ugly. Would that even be sufficient though? Conversely, would GCC developers be willing to agree that, when GCC detects such inconsistencies, it wouldn't take adverse advantage of it (e.g., by turning off LTO in this case, or similar)? Other possibilities for Autoconf would be to work toward a new set of checking macros (or extensions of current one) where the configure.ac author passes a full prototype for each function to check (Autoconf could keep a list of known prototypes for often-checked functions). I'm not sure how to fix the libtool symfile in a C99-conforming way. Thanks for reading this far. Cheers, Ralf