On 24/07/18 09:40, Fredrik Hederstierna wrote:
> Hi
>
> This is a general question to all you working with GCC benchmarking.
>
> I have been working with code benchmarks like CSiBE for ARM. From
> time to time unpredicted results appears where numbers gets worse by
> no reason.
>
> When looking into what could cause this unpredictable behaviour, I
> found that there are (at least for CSiBE), tons of code warnings that
> could cause unpredictable code generation like -Wuninitialized and
> -Wmaybe-uninitialized. Alot of these warning indicates real bugs.
>
> I added this issue in bugzilla, #Bug 85880 - "Different code
> generation for uninitialized variables" Though it got (correctly)
> rejected, since its not a bug. But still I'm thinking how this apply
> to benchmarking code, and to how to approach and address this fact.
>
> So my question is how to approach this problems when doing
> benchmarking, ofcourse we want the benchmark to mirror as near as
> 'real life' code as possible. But if code contains real bugs, and
> issues that could cause unpredictable code generation, should such
> code be banned from benchmarking, since results might be misleading?
> On the other hand, the compiler should generate best code for any
> input?
>
> What do you think, should benchmarking code not being allowed to have
> eg warnings like -Wuninitialized and maybe -Wmaybe-uninitialized? Are
> there more warnings that indicate unpredictable code generations due
> to bad code, or are the root cause that these are 'bugs', and we
> should not allow real bugs at all in benchmarking code?
>
> I've read some about uninitialized variable issues also at this link
> which was interesting, its a wider discussion, and ofcourse a very
> hard problem to solve.
> https://gcc.gnu.org/wiki/Better_Uninitialized_Warnings
>
> Thanks, Best Regards, Fredrik
>
AFAIUI, "-Wuninitialized" should not give false positives - if it
triggers a warning, it is because you are using a variable without
initialisation. The code is clearly wrong, and should not be used for
real code or for benchmarking.
"-Wmaybe-unitialized" indicates that there /could/ be a problem - it is
code that you need to look at carefully.
I'd say that in benchmarking code, you want to be very careful to
generate predictable and deterministic results - and that means no use
of uninitialised data. You need to get as close to perfectly repeatable
results as the hardware allows - only then can you know what changes the
performance of the code or hardware. If your results and timings depend
on what values happen to lie in memory before the benchmark runs, you
are guessing, not measuring.
If you get a "-Wuninitialized" warning, it's a bug in the code, and
needs fixed. A "-Wmaybe-uninitialized" warning may be a false positive,
so you might want to modify code to avoid such false positives (so that
you can see the real problems found by the warning). The gcc manual
gives this example:
{
int x;
switch (y)
{
case 1: x = 1;
break;
case 2: x = 4;
break;
case 3: x = 5;
}
foo (x);
}
A suggested fix is an "assert(0)" in the default case. You could also
have a "__builtin_unreachable()" instead - this will avoid any extra
run-time code, but of course you have to be very sure that x is valid or
you have serious undefined behaviour.
A lazy workaround would be to use "int x = x;".
