This is effectively v5 of the unittests proposal; for the earlier versions see: * v1: https://gcc.gnu.org/ml/gcc-patches/2015-06/msg00765.html * v2: https://gcc.gnu.org/ml/gcc-patches/2015-06/msg01224.html * v3: https://gcc.gnu.org/ml/gcc-patches/2015-10/msg02947.html * v4: https://gcc.gnu.org/ml/gcc-patches/2015-11/msg02379.html
Bernd said (in https://gcc.gnu.org/ml/gcc-patches/2015-11/msg01981.html ): > For some of the simpler infrastructure tests such as the ones in this > patch kit (bitmap, vec or wide-int functionality testing and such), > we had the idea of putting these into every ENABLE_CHECKING compiler, > and run them after building stage1, controlled by a -fself-test flag. > It's better to detect such basic failures early rather than complete > a full bootstrap and test cycle. It also keeps the tests alongside the > rest of the implementation, which I consider desirable for such > relatively simple data structures." So the main difference is this version of the patch kit is that the tests are run much earlier: rather than have a DejaGnu test below gcc.dg that compiles a dummy file with -fself-test, in this iteration, gcc/Makefile.in is updated so that the selftests are run during the build. The patch kit adds a "selftests" phony target to gcc/Makefile.in ensuring that the selftests must be successfully run for the build to complete. The time taken to run the testsuite is also printed. A "s-selftests" file is created, so that the selftests need only be run once each time "cc1" is rebuilt in a working copy. They are run at each stage in a bootstrap. For example, looking at a bootstrap log: $ grep "fself-test:" test/experiment/x86_64-pc-linux-gnu/build/make.log -fself-test: 522 pass(es); 0 failure(s) in 0.013000 seconds -fself-test: 522 pass(es); 0 failure(s) in 0.006000 seconds -fself-test: 522 pass(es); 0 failure(s) in 0.006000 seconds shows that the selftests are indeed run at each stage of a bootstrap, and that they're fast (well under a second). There's also a debug target, so that we can type: make selftests-gdb to run the selftests under gdb. I also patched things so that it aborts on the first failure, making failures easy to track down in the debugger, though requiring us to make the selftests robust. Assuming that other people like these changes, it has a few implications: * selftests ought to be fast. We don't want to slow down the build. * running -fself-test should be terse by default; we may want an option for it to be more verbose * poorly written selftests could break the build for other people (e.g. accidentally making target-specific assumptions). The patch kit adds some new tests: * some self-tests for diagnostic-show-locus.c. This shows how to unit-test an internal API. * conversion of the Levenshtein selftest from a plugin to using -fself-test (we could move various other tests from plugins to run earlier). There is a bug with GNU ld in which it doesn't run tests in source files that purely contain tests (hence I was #include-ing these from toplev.c). I tracked down the issue, which I've filed against GNU ld here: https://sourceware.org/bugzilla/show_bug.cgi?id=20152 and a minimal reproducer can be seen here: https://github.com/davidmalcolm/test-ctor The issue is that these don't work with GNU ld if they're linked into a .a file (libbackend.a in our case) and then linked from there into cc1. They work if they're linked directly into cc1 without going through an archive. We could link these files directly into cc1, or we could move to manual test registration instead. Or we could not have registration, and simply hardcode what tests run, and what order they run in. Testing: * successfully bootstrapped®ression tested on x86_64-pc-linux-gnu. * successfully built with 193 configurations using contrib/config-list.mk. The remaining configurations fail for various reasons: * 9 vxworks configurations fail to run the selftests with: xgcc: fatal error: environment variable ‘WIND_BASE’ not defined * 4 configurations fail due to pre-existing warnings: fails on 3 configurations (microblaze and rl78) due to warnings which I've posted fixes for), and on avr-rtemsOPT-enable-obsolete due to a different pair of warnings. * 2 failures: rs6000-ibm-aix4.3 and rs6000-ibm-aix5.1.0 due to a pre-existing bug on trunk (I've filed this as PR target/71375). * I haven't yet tested this with build != host. Patch 1 is the main part of the implementation; it's similar to v4. Patch 2 adds the new Makefile.in integration. Patch 3 reduces the verbosity of the tests by moving almost all logging to a new optional logfile (without providing a way to set it at this stage). Patch 4 shows a way of filtering the tests using a command-line regex. Patch 5 adds a new example test, showing how to unit-test an implementation detail of diagnostic-show-locus.c Patch 6 shows an example of porting one of our existing plugin-based tests to be within -fself-test instead. Patches 7-21 are the previously posted tests, albeit with some bug fixes. My recollection is that the major unresolved discussion areas in v4 of the patch kit were: (a) do we want to stop running at the first failure, or to run the testsuite to completion? (b) should tests be automatically registered, or should this be manual? (c) is this over-engineered; should we simply use gcc_assert etc? For (a), this version of the patch kit switches to stopping at the first failure. For (b), this version of the patch kit stays with auto-registration. For (c), this version stays with a "lite" version of GTest. Thoughts? Does the "running tests early" approach suggest answers to these? David Malcolm (21): Selftest framework (unittests v5) Makefile.in integration Various selftest::runner tweaks. Add -fself-test-regex= Add selftests for diagnostic-show-locus.c Convert Levenshtein test from a plugin to a selftest Add selftests to bitmap.c Add selftests to tree-cfg.c Add selftests to et-forest.c Add selftests to fold-const.c Add function-tests.c Fix warning in function-tests.c Fixup to function-tests.c Remove x86_64-isms in function-tests.c Add selftests to gimple.c Add hash-map-tests.c Add hash-set-tests.c Add selftests to input.c Add rtl-tests.c Add selftests to tree.c Add selftests to vec.c gcc/Makefile.in | 26 +- gcc/bitmap.c | 92 ++++ gcc/common.opt | 8 + gcc/diagnostic-show-locus.c | 113 ++++ gcc/et-forest.c | 99 ++++ gcc/fold-const.c | 66 +++ gcc/function-tests.c | 629 +++++++++++++++++++++++ gcc/gimple.c | 103 ++++ gcc/hash-map-tests.c | 81 +++ gcc/hash-set-tests.c | 57 ++ gcc/input.c | 110 ++++ gcc/rtl-tests.c | 98 ++++ gcc/selftest.c | 210 ++++++++ gcc/selftest.h | 267 ++++++++++ gcc/spellcheck.c | 48 ++ gcc/testsuite/gcc.dg/plugin/levenshtein-test-1.c | 9 - gcc/testsuite/gcc.dg/plugin/levenshtein_plugin.c | 64 --- gcc/testsuite/gcc.dg/plugin/plugin.exp | 1 - gcc/toplev.c | 55 ++ gcc/toplev.h | 2 + gcc/tree-cfg.c | 264 ++++++++++ gcc/tree.c | 47 ++ gcc/vec.c | 142 +++++ 23 files changed, 2512 insertions(+), 79 deletions(-) create mode 100644 gcc/function-tests.c create mode 100644 gcc/hash-map-tests.c create mode 100644 gcc/hash-set-tests.c create mode 100644 gcc/rtl-tests.c create mode 100644 gcc/selftest.c create mode 100644 gcc/selftest.h delete mode 100644 gcc/testsuite/gcc.dg/plugin/levenshtein-test-1.c delete mode 100644 gcc/testsuite/gcc.dg/plugin/levenshtein_plugin.c -- 1.8.5.3
