Question about GCC benchmarks and uninitialized variables
Hi This is a general question to all you working with GCC benchmarking. I have been working with code benchmarks like CSiBE for ARM. >From time to time unpredicted results appears where numbers gets worse by no >reason. When looking into what could cause this unpredictable behaviour, I found that there are (at least for CSiBE), tons of code warnings that could cause unpredictable code generation like -Wuninitialized and -Wmaybe-uninitialized. Alot of these warning indicates real bugs. I added this issue in bugzilla, #Bug 85880 - "Different code generation for uninitialized variables" Though it got (correctly) rejected, since its not a bug. But still I'm thinking how this apply to benchmarking code, and to how to approach and address this fact. So my question is how to approach this problems when doing benchmarking, ofcourse we want the benchmark to mirror as near as 'real life' code as possible. But if code contains real bugs, and issues that could cause unpredictable code generation, should such code be banned from benchmarking, since results might be misleading? On the other hand, the compiler should generate best code for any input? What do you think, should benchmarking code not being allowed to have eg warnings like -Wuninitialized and maybe -Wmaybe-uninitialized? Are there more warnings that indicate unpredictable code generations due to bad code, or are the root cause that these are 'bugs', and we should not allow real bugs at all in benchmarking code? I've read some about uninitialized variable issues also at this link which was interesting, its a wider discussion, and ofcourse a very hard problem to solve. https://gcc.gnu.org/wiki/Better_Uninitialized_Warnings Thanks, Best Regards, Fredrik
Re: Question about GCC benchmarks and uninitialized variables
On Tue, 24 Jul 2018, Fredrik Hederstierna wrote: > So my question is how to approach this problems when doing benchmarking, > ofcourse we want the benchmark to mirror as near as 'real life' code as > possible. But if code contains real bugs, and issues that could cause > unpredictable code generation, should such code be banned from benchmarking, > since results might be misleading? Well, all benchmarks are going to be imperfect reflections of real-life workloads in the first place, so their bugs just increase the degree to which they are misleading. When a new compiler version starts to treat some undefined piece of code differently, it can cause a range of effects from code size perturbations as in your case, to completely invalidating the benchmark as in spec2k6 x264 benchmark's case (where GCC exploited undefined behavior in a loop, turning it to an infinite loop that eventually segfaulted). Perhaps even though results on individual benchmarks can vary wildly, aggregated results across a wide range of non-toy benchmarks may be indicative of ... something, because they are unlikely to all exhibit the same "bugs". > On the other hand, the compiler should > generate best code for any input? Engineering effort is limited, so it's probably better to go for generating good code for inputs that are likely to resemble actively used code (and in actively used&maintained code, bugs can be reported and fixed) :) > What do you think, should benchmarking code not being allowed to have eg > warnings like -Wuninitialized and maybe -Wmaybe-uninitialized? Are there more > warnings that indicate unpredictable code generations due to bad code, or are > the root cause that these are 'bugs', and we should not allow real bugs at all > in benchmarking code? A blanket ban on warnings won't work, they have false positives (especially the -Wmaybe- one), and there exist code that validly uses uninitialized data. I don't have such a striking example for scalar variables, but for uninitialized arrays there's this sparse set algorithm (which GCC itself also uses): https://research.swtch.com/sparse I think good benchmarks sets should be able to evolve to account for newly discovered bugs, rather then remain frozen (which sounds like a reason to become obsolete sooner rather than later). Alexander
Re: Question about GCC benchmarks and uninitialized variables
On 24/07/18 09:40, Fredrik Hederstierna wrote:
> Hi
>
> This is a general question to all you working with GCC benchmarking.
>
> I have been working with code benchmarks like CSiBE for ARM. From
> time to time unpredicted results appears where numbers gets worse by
> no reason.
>
> When looking into what could cause this unpredictable behaviour, I
> found that there are (at least for CSiBE), tons of code warnings that
> could cause unpredictable code generation like -Wuninitialized and
> -Wmaybe-uninitialized. Alot of these warning indicates real bugs.
>
> I added this issue in bugzilla, #Bug 85880 - "Different code
> generation for uninitialized variables" Though it got (correctly)
> rejected, since its not a bug. But still I'm thinking how this apply
> to benchmarking code, and to how to approach and address this fact.
>
> So my question is how to approach this problems when doing
> benchmarking, ofcourse we want the benchmark to mirror as near as
> 'real life' code as possible. But if code contains real bugs, and
> issues that could cause unpredictable code generation, should such
> code be banned from benchmarking, since results might be misleading?
> On the other hand, the compiler should generate best code for any
> input?
>
> What do you think, should benchmarking code not being allowed to have
> eg warnings like -Wuninitialized and maybe -Wmaybe-uninitialized? Are
> there more warnings that indicate unpredictable code generations due
> to bad code, or are the root cause that these are 'bugs', and we
> should not allow real bugs at all in benchmarking code?
>
> I've read some about uninitialized variable issues also at this link
> which was interesting, its a wider discussion, and ofcourse a very
> hard problem to solve.
> https://gcc.gnu.org/wiki/Better_Uninitialized_Warnings
>
> Thanks, Best Regards, Fredrik
>
AFAIUI, "-Wuninitialized" should not give false positives - if it
triggers a warning, it is because you are using a variable without
initialisation. The code is clearly wrong, and should not be used for
real code or for benchmarking.
"-Wmaybe-unitialized" indicates that there /could/ be a problem - it is
code that you need to look at carefully.
I'd say that in benchmarking code, you want to be very careful to
generate predictable and deterministic results - and that means no use
of uninitialised data. You need to get as close to perfectly repeatable
results as the hardware allows - only then can you know what changes the
performance of the code or hardware. If your results and timings depend
on what values happen to lie in memory before the benchmark runs, you
are guessing, not measuring.
If you get a "-Wuninitialized" warning, it's a bug in the code, and
needs fixed. A "-Wmaybe-uninitialized" warning may be a false positive,
so you might want to modify code to avoid such false positives (so that
you can see the real problems found by the warning). The gcc manual
gives this example:
{
int x;
switch (y)
{
case 1: x = 1;
break;
case 2: x = 4;
break;
case 3: x = 5;
}
foo (x);
}
A suggested fix is an "assert(0)" in the default case. You could also
have a "__builtin_unreachable()" instead - this will avoid any extra
run-time code, but of course you have to be very sure that x is valid or
you have serious undefined behaviour.
A lazy workaround would be to use "int x = x;".
Important Inquiry
Hello We have a client who is interested in some of your services. I will provide further details should we get a response from you. Kind regards Mr J. Hamilton CEO Hamtons Merchants Trading Co. Int'l
Re: GCC 8.2 Status Report (2018-07-19), branch frozen for release
On Thu, Jul 19, 2018 at 10:11 AM, Richard Biener wrote: > > Status > == > > The GCC 8 branch is frozen for preparation of the GCC 8.2 release. > All changes to the branch now require release manager approval. > > > Previous Report > === > > https://gcc.gnu.org/ml/gcc/2018-07/msg00194.html Is there any chance we can get some of the spectrev1 mitigation patches reviewed and into 8.2 . It would be quite useful to get these into a release as I see that the reviews are kinda petering out and there hasn't been any objection to the approach. regards Ramana
Re: GCC 8.2 Status Report (2018-07-19), branch frozen for release
On July 24, 2018 5:50:33 PM GMT+02:00, Ramana Radhakrishnan wrote: >On Thu, Jul 19, 2018 at 10:11 AM, Richard Biener >wrote: >> >> Status >> == >> >> The GCC 8 branch is frozen for preparation of the GCC 8.2 release. >> All changes to the branch now require release manager approval. >> >> >> Previous Report >> === >> >> https://gcc.gnu.org/ml/gcc/2018-07/msg00194.html > >Is there any chance we can get some of the spectrev1 mitigation >patches reviewed and into 8.2 . It's now too late for that and it has to wait for 8.2. >It would be quite useful to get these into a release as I see that the >reviews are kinda petering out and there hasn't been any objection to >the approach. It's not that people only use release tarballs. Richard. > > >regards >Ramana
