Rather that starting from scratch, you should strongly consider adapting something like test-framework to this task, as it already has done the heavy work of creating a way to combine tests from different frameworks into a single suite and includes such features as displaying a progress bar during the QuickCheck tests. Furthermore, it is easily extendable to support new kinds of tests; for example, I found that it was relatively straightforward to add a new kind of "statistical" test to make sure that the average value of a function where where it should be.
Cheers, Greg On Apr 6, 2010, at 3:51 PM, Thomas Tuegel wrote: > Hello again! > > Based on the invaluable feedback I've received, I've made some > revisions to the proposal I made a few days ago (at the end of this > post, after my signature). I apologize for the length of my post, but > I'd like once again to solicit feedback on this. Any commentary is > very helpful! > > Thanks! > -- > Thomas Tuegel > > > Throughout this proposal, examples are given to indicate how a package > author would utilize the features proposed here. In all these > examples, suppose that the programmer is the author of the > 'haskell-foo' package, which exposes the module 'Foo' and has a single > test executable, 'foo-tests', using the QuickCheck testing library. > > > Package Description File Syntax > > The syntax for designating test executables in package description > files will be based on the existing syntax for describing executables. > Such a stanza in the hypothetical package's description file would > look like: > >> Test foo-tests >> main-is: foo-tests.hs >> build-depends: haskell-foo, Cabal, QuickCheck > > This example is obviously minimal; this is really an 'Executable' > stanza by another name, so any options recognized there would also be > valid here. > > > Handling of Test Executables by Cabal > > The changes proposed here will make it possible to build, test, and > install a Cabal package with the usual sequence of commands: > > $ cabal configure > $ cabal build > $ cabal test > $ cabal install > > Cabal will recognize two new options during the 'configure' stage: > '--enable-tests' and '--disable-tests'. > > If 'cabal configure' is invoked with the '--enable-tests' option, then > any test executables designated in the package description file will > be built. For the purposes of the 'configure' and 'build' stages, > they will be handled as if they were ordinary executables, i.e., > described by 'Executable' stanzas. With tests enabled, the test > programs will be executed and their results collected by Cabal during > the 'test' stage. > > If 'cabal configure' is invoked with the '--disable-tests' option > (which should be the default if neither option is specified), then > test executables designated in the package description file will be > ignored, as if the 'Test' stanza were absent. Any attempt to invoke > the 'test' stage with tests disabled should remind the user of that > fact. > > Regardless of the status of tests (enabled or disabled), the 'install' > stage will ignore any executables designated as test suites, since it > is not desirable to install the test executables. > > > Collection of Test Results > > Cabal will provide a standard interface, residing in the module > 'Distribution.Test', for running tests independent of the testing > library used. A minimal outline of this module looks like: > >> module Distribution.Test where >> >> type Name = String >> type Result = Maybe Bool >> type Info = String >> type Output = String >> >> -- 'Compiler' and 'ComponentLocalBuildInfo' are already provided by Cabal. >> -- They are included here to aid in debugging test failures >> type Report = (Compiler, ComponentLocalBuildInfo, [(Name, Result, Info, >> Output)]) >> >> class Test t where >> wrap :: t -> IO (Result, Info) >> >> runTests :: Test t => [(Name, t)] -> IO Report >> >> writeResults :: Report -> IO () > > Instances of 'Test' will run a type of test from one of the testing > libraries; part of this project will therefore be patching QuickCheck > and HUnit to provide these instances. Any other testing library > providing this instance will also be compatible with the automated > testing features this proposal introduces. > > The type 'Maybe Bool' is used throughout this framework to indicate a > test result: 'Nothing' indicates a test was not run, 'Just False' > indicates a failed test, and 'Just True' indicates a successful test. > The 'Info' string captures any information provided by the testing > library. However, because of the reliance of most test suites on > standard output, Cabal will also capture the standard output produced > during each test (when the test suite is invoked through 'cabal > test'); the output will be included in the test result file. > > The function 'writeResults' will write the test results to a file. > The 'Show' instance for the type of its single argument will therefore > constitute the standard test result file format. This has the > advantage of being human- and machine-readable without requiring any > extra dependencies to parse the file. > > With this framework, the hypothetical package's author might write a > test suite such as: > >> module Main where >> >> import Distribution.Test >> import Foo >> import QuickCheck >> >> testBar :: Gen Bool >> testBar = ... >> >> testBaz :: Gen Bool >> testBaz = ... >> >> main = runTests [("testBar", testBar), ("testBaz", testBaz)] >>= writeResults > > > Reporting and Comparing Test Results > > The 'cabal test' command will run tests by default, but support two > other options: > > 1. '--report [file]', which will produce a nicely formatted > report of the test results stored in the named file, or of the last > run of the package's test suite if no file is specified, and > 2. '--diff file1 file2', which will show the differences between > the test results stored it two different files. > > Because the report file format is readily parseable by any Haskell > program, it could be processed into another format for compatibility > with existing tools. > _______________________________________________ > Haskell-Cafe mailing list > Haskell-Cafe@haskell.org > http://www.haskell.org/mailman/listinfo/haskell-cafe _______________________________________________ Haskell-Cafe mailing list Haskell-Cafe@haskell.org http://www.haskell.org/mailman/listinfo/haskell-cafe