During implementing my previously mentioned test history recorder, I discovered an issue with the current FailureRecorder implementation. The problem I'm seeing is that it doesn't write the failure suite Java class when the JUnitTask forks a new JVM to execute tests. FailureRecorder relies on registering itself as a BuildListener by overriding ProjectComponent's setProject() method. JUnitTestRunner doesn't have access to the current Project since it's running in a separate JVM and so never invokes setProject(). This means that the taskFinished() method is never invoked either and the recorded results are lost.
Once I gave this some thought, it seemed to me that FailureRecorder was doing too much to be thought of as just a formatter. I'm not sure it should even be declared in the build.xml using the <formatter/> tag since it's not formatting anything but that's another issue. I was following the same method of hooking the taskFinished() method for my HistoryRecorder but had to make some changes in order to work in a forked JVM. At this point my solution was to add two new lifecycle methods to the JUnitResultFormatter interface: startAllTestSuites() and endAllTestSuites(). The JUnitTestRunner then invokes these methods and allows the recorder to write it's results. This solution has advantages and disadvantages. To the good, it removes the need for the FailureRecorder to extend ProjectComponent and simplifies the implementation (also makes it work in a forked JVM!). On the bad side, it complicates JUnitTestRunner and the JUnitResultFormatter interface. This could be cleaned up by adding a new "recorder" concept rather than heavily overloading the existing "formatter" tag. Another unfortunate side effect of having recorders masquerade as formatters is the need for all FailureRecorder instances communicate via a static Set of failed tests (JUnitTestRunner creates a fresh formatter instance for each test suite). This works and I used the same mechanism for my HistoryRecorder but it doesn't feel quite right to me. Should I open a bug report for the FailureRecorder in a forked JVM problem? Thanks, -Clark A.
