On 10/27/2015 01:48 PM, David Malcolm wrote:
This is a followup to these proposals:
* v1: https://gcc.gnu.org/ml/gcc-patches/2015-06/msg00765.html
* v2: https://gcc.gnu.org/ml/gcc-patches/2015-06/msg01224.html
The following patch kit adds a unit tests framework for gcc,
as a new subdirectory below gcc/testsuite.
The aim is to give us direct coverage of low level implementation
details that aren't well covered by our existing tests, such as
our data structures, gengtype, etc. This will let us have tests
that directly poke at specific internal APIs and verify that the
right thing happens, rather than having to write source code and
option combinations that we hope touches the relevant state.
It should make it trivial to add new unit tests.
I see it as complementary to our existing test approaches.
Right. And I feel I ought to chime in since I've been the source of
confusion WRT unit testing. I have often mis-used the term "unit
testing" to mean feed state/code into a pass and test what comes out.
But that's not actually unit testing and not the focus of this patchkit
from David. As David states, the idea here is to get direct testing
coverage of low level internals. I've been pondering this problem a
fair amount lately as I look at our codebase and ponder how to improve
its maintainability over time.
This kind of low level testing has value. I don't doubt that in the
slightest anymore. I fear the real problem in this space is getting
enough scaffolding around those low level things we want to test so that
we can write tests around them. I wouldn't at all be surprised if we go
through multiple iterations on the overall design & implementation.
Hell, you're already on your 3rd :-)
Like previous versions of the patch kit it uses the Google Test
framework, since I've had good experiences with it:
http://code.google.com/p/googletest/
and like v2 of the kit it embeds a two-file copy of v1.7 of
the kit, to avoid adding extra dependencies (one .h file and one
.c file).
v1 of the kit was structured as a frontend, v2 of the kit as
a plugin that was built as if it were a frontend. Both of
these approaches were problematic, so this version
of the patch kit simply builds as a test case within the
plugin.exp suites.
Using a plugin approach allows us to avoid at least some of the problems
in getting a test that can be built because ultimately the bits get
sucked into the address space of the compiler as a DSO.
I suspect that's the only sensible model in the short/medium term. I
just don't see us getting to a point where (for example) we could
instantiate some internal class as an independent test totally
independent of the rest of the compiler without a ton more work.
I will note that large blobs of what Andrew has been doing would be a
prerequisite for getting to that point.
Anyway, it's a bit of rambling, but in the end, I think the summary is
the plugin approach seems right for where we are today and expect to be
for a while.
In this approach, each of gcc.dg and g++.dg have a plugin.exp that
lists plugins to be built and source files to then compile with
the plugin. This patch kit inserts into them references to the plugin
within ../../unittests, so that we can run the tests within both cc1
and cc1plus without having to list everything twice. The plugin
is built by plugin.exp and plugin-support.exp as per the existing
test cases. Doing so requires #include-ing everything relevant
within unittests-plugin.c.
So as I ponder how we'll use this in practice, I think we'll probably
need a way to control which tests run with a fair amount of granularity.
So for example, if I'm doing something like cleaning up the SSA_NAME
manager, I'd really like to be able to unit test that and do so quickly
& repeatedly during development before moving on to a larger test like
bootstrap & regression test. Similarly if I had this framework in place
earlier, I would have wanted run tests around just the scoped hashtable
interface and implementation.
Conceptually this might work in the same manner we've done with
c-torture & the dg frameworks. ie
RUNTESTFLAGS=unittest.exp=scopedtables
If you've already got that capability in place, then well, that's awesome.
The plugin uses a custom gtest reporter that reports results in
DejaGnu format. The kit adds some logic to testsuite/lib/prune.exp
to parse the DejaGnu output on stderr e.g.:
PASS: ggc_test.inheritance
and re-emits it at the Tcl level, so that the unit test results from
the plugin are reported within the regular test output like this:
PASS: gcc.dg/plugin/unittests.c -fplugin=./unittests-plugin.so
ggc_test.inheritance
I hate to note it, but isn't the pruning code fairly performance
sensitive? Any way to do this outside of prune.exp that doesn't effect
running the rest of the testsuite? Or show that it just "doesn't
matter" having these extra bits in the pruning code.
Successfully bootstrapped®rtested on x86_64-pc-linux-gnu
(on top of r229364); adds 61 new PASS results to each of gcc.sum and
g++.sum.
I've broken up the patches into logical chunks in the hope of making
review easier, but this is effectively one change, and would be
committed as such. Though if any individual testcases are
problematic, they could be deferred, if that will help get the rest
of the work approved.
You know me. I always try to carve off the easiest parts of the review
first, then iterate on what's left. It's usually the case that each
iteration makes something in the next iteration easier to understand.
So you should expect reviews to arrive in a semi-random order.
[There are some FIXMEs in the testcases indicating room for further
testing, and I've not fully reduced the #includes in them; is the
"blessed" #include order documented yet, and do we have tooling
in "contrib" for this yet?]
The tooling isn't finished, but the last patch from Andrew has usable
versions. I've used show-headers extensively, but order-headers and
reduce/replace seem to work when I've done light poking. I'd certainly
recommend using them for the reduction.
Jeff
