On 10/29/2015 10:10 AM, David Malcolm wrote:
There may be useful things in gtest for us that I'm not using yet. For
example, the support for death tests may be useful for testing that e.g.
our checking macros kill the program sanely in the presence of malformed
internal data.
Please check on this. I want to build a set of tests around the
SSA_NAME manager, and that would include setting up can't happen
situations and seeing if they're properly diagnosed by the checking
code. So obviously I have an interest in how the framework handles this
kind of scenario.
I wanted to reuse the gtest code on the grounds that:
* the code exists
* it's maintained (https://github.com/google/googletest
shows the last commit was 10 days ago)
* it has documentation:
https://github.com/google/googletest/blob/master/googletest/docs/Primer.md
(there's much more that just that)
* it will be familiar to some people from other projects
That said, if it's a blocker, I can attempt a minimal reimplementation
of the subset of the API that I'm using.
I'd probably lean towards continuing to use gtest. I don't currently
see a compelling reason to build another framework.
v1 of the kit was structured as a frontend, v2 of the kit as
a plugin that was built as if it were a frontend. Both of
these approaches were problematic, so this version
of the patch kit simply builds as a test case within the
plugin.exp suites.
This is the part I'm least certain about, for two reasons:
* They are more like source files than tests, and I think I'd prefer
to have them alongside the source, in gcc/ rather than in the
testsuite. This way people are invariable going to fail to notice
them when they grep for something.
So something like:
gcc/test-vec.c
or even:
gcc/vec-tests.c
so that it sorts next to gcc/vec.h in a listing?
i.e. gcc/FOO-tests.c for gcc/FOO.c
Maybe the unittests-plugin.c should live within the "gcc" directory?
It's a bit of bike-shedding :-) I'd be happy with gcc/test-whatever or
gcc/whatever-test. Both have pluses/minuses and in the end I doubt it'd
matter much.
My only worry here is doubling the number of files in gcc/. But I'd
claim that we want to have more subdirectories for major logical areas.
gimple/ rtl/ whatever, which would significantly mitigate the problem
of too many files in gcc/
* This uses a plugin into whatever compiler was built, but sometimes
you can't really set up unit tests that way because what you want
to test depends on target specifics. What I've often wanted is a
special test target that gets built with a special machine
description that has whatever patterns are needed to replicate
tricky situations in reload or other optimization passes.
The tests you have so far are focused mostly on high-level gimple/tree
tests where this limitation is probably not showing up very much, but I
think it would be better to have something that allows us to have more
in-depth tests.
I think our long-term goal should be at least 5 styles of test:
(A) end-to-end tests of the compiler: running it on some source and
verifying properties of the diagnostics and/or generated code
(potentially running it). This is what we have today.
(B) unit tests of individual subsystems: tests that exercise specific
internal APIs e.g. containers like vec<>, hash_map<> etc, also things
like gengtype. This is a gap in our current test coverage, and what
this patch kit aims to start filling.
Right. It's essentially data structure and other very low level testing
(gengtype).
That's even true if I think about the SSA_NAME manager -- in an ideal
world, it'd just be a container for objects that we want to release &
recycle, whether they're SSA_NAMEs, pseudos, whatever we decide has the
right properties to benefit from recycling. The fact that it's
recycling SSA_NAMEs isn't for the most part all that important.
(C) and (D): tests of specific passes: tests that accept IR (either
gimple or RTL), and run just one pass, and verify properties of the IR
emitted by the pass, or messages emitted by the pass. LLVM has a "-R"
remark option, so that a pass can issue remarks about what it's doing (a
new kind of diagnostic):
http://llvm.org/releases/3.5.0/tools/clang/docs/UsersManual.html#opt-rpass
We could implement this kind of testing by implementing gimple and RTL
frontends, and a -R option, which could integrate nicely with DejaGnu,
with
/* { dg-remark "inlined here" } */
and the like.
This would require finishing the gimple FE, and writing an RTL FE
(independent goals). Hopefully that would give us test coverage for
dealing with e.g. tricky reload situations.
Insert note from Richi (was it?) where he wants the gimple FE made real
and integrated. I think you'll have technical support if you want to
tackle that next year.
Something similar for RTL would be cool, but probably even harder given
the amount of state that's traditionally been kept out of the IL stream.
I'm sure some things are better today than in the past, but it's
probably a very tanged mess to get to where we want to go with RTL.
jeff
