On Mon, Nov 21, 2005 at 06:22:37PM +0000, Ian Jackson wrote: > > Note that it's often better to have a single script run many tests, so > > you probably want to allow tests to pass back some summary information, > > or include the last ten lines of its output or similar. Something like: > > foo FAIL: > > FAILURE: testcase 231 > > FAILURE: testcase 289 > > FAILURE: testcase 314 > > 3/512 test cases failed > This is no good because we want the test environment to be able to > tell which tests failed, so the test cases have to be enumerated in > the test metadata file.
Uh, having to have 1000 scripts, or even just 1000 entries in a metadata
file, just to run 1000 tests is a showstopper imho. Heck, identifying
testcases by number mightn't be particularly desirable in some instances,
if a clearer alternative like, say, "test case failed: add 1, add 2,
del 1, ch 2" is possible.
> You can't check that the binary works _when the .deb is installed_
> without installing it.
That's okay. You can't check that the binary works _on the user's system_
without installing it on the user's system either. For Debian's purposes,
being able to run the tests with minimal setup seems crucial.
> Also, a `Restriction' isn't right because if the test has neither of
> those Restrictions then presumably it can do either but how would it
> know which ?
It would have to not care which; which it could do by expecting the
test harness to put the binaries in the PATH, or provide an environment
variable like INSTALL_ROOT=$(pwd)/debian/tmp .
> No, I mean that if the tests live (say) in
> build/foo-1.0/debian/tests/x then build/foo-1.0/debian/tests/control
> could say
> Depends: bar
> which would mean bar would have to be installed, effectively making it
> an integration test.
Having test case dependencies is fairly useful; in any event the language
"Even integration tests can be represented like this: if one package's
tests Depend on the other's" is wrong if tests depend on other packages,
not on other package's tests. You'll want Conflicts: as well as Depends:
in that case too.
It would probably be quite useful to be able to write tests like:
for mta in exim4 smail sendmail qmail; do
install $mta
# actual test
uninstall $mta
done
too, to ensure that packages that depend on one of a number of packages
actually work with all of them.
Cheers,
aj
signature.asc
Description: Digital signature
