On Wed, 2010-04-07 at 16:09 -0400, Thomas Tuegel wrote: > On Wed, Apr 7, 2010 at 3:33 PM, Duncan Coutts > <duncan.cou...@googlemail.com> wrote: > > The importance of this is that it lets us develop improved testsuite > > interfaces in future. At the moment there are two test interfaces we > > want to support. One is the simple unix style "exit code + stdout" > > interface. This is good because it is a lowest common denominator that > > all existing testsuites can fit into. > > > > Of course that test interface does not provide any detailed > > machine-readable information (though you do get human-readable test > > logs). So that's why we want a second interface. That one should let the > > "testing agent" (for example "cabal test" but could be other agents) get > > much more detail about what tests can be run and then what the results > > are of various tests. > > For the purpose of differentiating between these two, would a field in > the test section such as "interface: stdout" (in the first case) or > "interface: detailed" (in the second) suffice?
Yep. > > The details of such an interface are up for discussion. I do not mind if > > that is a command line executable interface or a library interface. > > That's something I've been thinking about. The former seems more > portable. Maybe cabal could call the test program with "test_program > --list" to produce a list of tests and "test_program > test1,test2,test3" to run some tests. Having some ideas is good. The details of the interface can be worked out during the project. > I also want to ask how strictly this is within the scope of the SoC > project, i.e. how well will my proposal be received if it focuses > primarily on the first part of the problem (getting everything working > for the stdout interface)? I ask because the detailed interface seems > like a much larger mandate given that cabal doesn't really support any > of the syntax/features for the simple stdout interface. Certainly we want to get the first part working first. My guess is that there is time within the 3-month GSoC period to complete the basic bits and to at least have a go at a prototype of a more detailed interface. That part doesn't need to be final, especially given that it will be possible to create new interfaces in future. > >> Handling of Test Executables by Cabal > >> > >> The changes proposed here will make it possible to build, test, and > >> install a Cabal package with the usual sequence of commands: > > > > That all sounds reasonable. I'd like to add that the interface between > > the testsuite and a testing agent such as the cabal program should be > > clearly documented and specified. There will likely be dedicated test > > agents that want to run the tests too and send reports to other systems > > (e.g. dedicated hackage test agents) and convert to other formats (e.g. > > integration in companies in-house systems). > > Rogan mentioned possible upcoming support in test-framework for JUnit > XML as an output format for test results. That certainly seems to be > widely supported; do you think it is suitable? I think it's important to be able to convert into standard or custom formats. I've no idea if JUnit XML would make sense as the native format. It's plausible. Duncan _______________________________________________ Haskell-Cafe mailing list Haskell-Cafe@haskell.org http://www.haskell.org/mailman/listinfo/haskell-cafe
