On Thu, 31 Mar 2011, Ralf Wildenhues wrote: > Hello, > > * Rainer Orth wrote on Thu, Mar 31, 2011 at 06:12:31PM CEST: > > Ian Lance Taylor <i...@google.com> writes: > > > Argh, no, I am trying to fight against that as long as possible. We > > > should be moving away from DejaGNU, not toward it. > > > > Do you have a decent alternative? I've no idea what happened to QMTest > > which CodeSourcery tried for the C++ testsuite. > > FWIW, Automake's parallel-tests driver at least provides halfway decent > test results summary; timeouts and remote control would need to be > implemented by means of a per-test driver script or program though. > And of course the summary style is different from the dejagnu one ...
As far as I could see when I last looked at it, Automake's test code lacks the basic feature of being able to test installed programs without reconfiguring and rebuilding the code. No-one really uses GCC with long series of -B options in a build tree; installed testing is much closer to how it's actually used and the aim should be to move towards staged installation for build-tree testing: install in a temporary directory within the build tree and run programs from there. The vast bulk of GCC users are probably using a version from a distributor who did install in a temporary directory at some point in packaging - the case of someone wanting to know if a tool is broken before running "make install" to replace a previous installation is not the usual one, and "make install" itself should just be copying from such a staging directory in the build tree. Basic features for a new test harness for GCC would include, in my view: * Starting from installed testing as the basic case (run a test-running command, such as runtest at present, to run the tests for the tools in the PATH), with more complicated commands for build-tree testing meaning passing a more complicated configuration file (generated by the build process) to the test-running command. Definitely do not hardcode details of build-tree structure in the test tool itself as DejaGnu does. * Well-defined enumeration of test assertions independent of running the tests. In DejaGnu the test assertions are the things after "PASS:" or "FAIL:" - but there can be duplicate names, and the set of names can depend on the results of the tests, or on the build directory name, when tests or testsuites are buggy; the system should make it inherently impossible to have such problems, so that it is always possible to examine results and see what tests did not run, for example. Similarly, test assertion names should be as stable as possible to facilitate comparisons when the testsuite itself as well as the compilers being testing may have changed. * Structured results so that build commands and output etc. are unambiguously associated with a particular test. (For DejaGnu you need to do heuristic processing of .log files.) * Well-defined interfaces for defining a target board (i.e., how you run a test program on the target, get its output, copy inputs to the board, etc.), a host board (for testing the tools themselves on a remote host), a testsuite. * Ability to represent expectations for test results just as flexibly as the present XFAILs system. Some test systems seem to be based on a presumption that the expected state is no failures at all, and this is not realistic for GCC. -- Joseph S. Myers jos...@codesourcery.com
