Request For Enhancement
The page: http://gcc.gnu.org/install/build.html does provide a few tips about
making a profiled bootstrap but I've found little info for coverage testing.
One would think that you could enable it in the ./configure script and type
make (Shhh: Andrew lets out that it won't work that way) ... why would we
read and follow the docs when there is a "secret way" (that is broken) for
us to use.
So I set out to devise a fast / simple / effective way to make the "C"
compiler retain coverage info for itself while the Testsuite ran.
# ../gcc_trunk/configure --enable-languages=c
# make
# make check
# make install
Now edit ../gcc_build/Makefile and replace all "-g -O2"'s with this:
"-g -O0 -fprofile-arcs -ftest-coverage". (Like the ./configure script
ought to do).
# echo The next command will need a trick to rebuild Ada (if used) afterwards.
# make clean-gcc
# make gcc
Since the build does not end after gcc is built _IF_ you enable Ada as one
of the languages you will need to come back and CTRL-C the build during
the making of the Ada portion. You will only need gcc and the "C" compiler
for these tests (but these instructions are applicable to all the compilers).
Now you have an installed "normal" compiler and a freshly built non-installed
"coverage statistic recording" compiler (for the "C" language only).
I can type this in the build directory:
# make -i -k check-gcc RUNTESTFLAGS="-v gcc.dg/graphite/graphite.exp"
...(Lots of wonderful output)
=== gcc Summary ===
# of expected passes 66
# of expected failures 2
That takes a while to run. I estimate it would take me a month to do
the whole Testsuite on my System - so I won't be the one to fix this!
I (and _most_ people) can afford the time and resources to do a _single_
test and ensure that _their_ Testsuite properly checks _their_ code
_before_ they submit it.
Now that I have used the Graphite Testsuite lets see how well it worked:
# cd ../gcc_build/gcc/
# /usr/local/bin/gcov -a -b -n -l -f -u graphite | ggrep executed:0.00 | wc -l
60
There are 60 functions with 0% of their code executed (not tested!).
# /usr/local/bin/gcov -a -b -n -l -f -u graphite
Here is a _small_ portion of the output:
File '../../gcc_trunk/gcc/graphite.h'
Lines executed:70.97% of 62
Branches executed:89.29% of 112
Taken at least once:56.25% of 112
No calls
File '../../gcc_trunk/gcc/graphite.c'
Lines executed:73.50% of 2385
Branches executed:79.90% of 1418
Taken at least once:62.76% of 1418
No calls
The analysis:
We test little more than 70% of our code.
We execute branches a bit better than 80% of the time and usually take them.
IE: Not great.
There is lots to be gained by this type of analysis and many different
ways to process the data (once you have it) to find out new answers.
We have a Testsuite to check for (apparently in some case perhaps only
70% of the) bugs but we need the companion "Testsuite Tester" to see if
the tests "run the code".
Note: "Running The Code" does NOT count as a PASS but NOT running the
code _does_ count as a FAIL. We have 1/3 of Graphite un-tested by our
current Testsuite. How do I know that for sure ... Because the code was
not even executed, let alone _properly_ tested.
A Makefile target (or script) that would allow someone with a dual-quad
to check the whole Toolchain once a week and ensure that gcc was built
and tested correctly would go a long way towards future maintenance ease.
Thanks for reading this,
Rob
--
Summary: RFE - Need Makefile to test coverage of Testsuite
Product: gcc
Version: 4.4.0
Status: UNCONFIRMED
Severity: enhancement
Priority: P3
Component: testsuite
AssignedTo: unassigned at gcc dot gnu dot org
ReportedBy: rob1weld at aol dot com
GCC build triplet: *-*-*-*
GCC host triplet: *-*-*-*
GCC target triplet: *-*-*-*
http://gcc.gnu.org/bugzilla/show_bug.cgi?id=38833
