Note: that this is one of two messages on roughly the same topic. This message will deal solely with TECHNICAL issues. If you have some technical followup then please go ahead and reply here. If you have a complaint or comment about my or Ubuntu's approach, please reply in debian-*project* to my other message, titled `Automated testing - politics, information, and Ubuntu's plans'.
(Also, yesterday I posted this message to -project rather than -policy by mistake. I will follow up to all of the replies there and try to move the discussion here.) The first two paragraphs are common to the two messages: One thing that the Debian universe is lacking is a good way to automatically test packages. This makes it hard to spot regressions, and difficult to be systematic. Ubuntu are proposing to invent a system to allow us to do automated regression tests. The full plan consists of a number of pieces and can be read here: https://wiki.ubuntu.com/AutomatedTesting If you're interested in this subject I'd appreciate it if you could go and take a look. Feel free to make your comments here on -policy, or any other appropriate channel. One of the important parts needed here is an interface for packages to supply tests. That is, a standard way for a Debian-format package to enumerate and describe the tests it supplies, and to allow a test harness to invoke them. Note that the point is to be able to test the _actual package_, _as installed_ (eg on a testbed system). This is much better than testing the package from the source treeu during build time because it will detect packaging mistakes as well as program source problems and as we know packaging mistakes of one kind or another are one of the main causes of problems. It seems to me that the right way for packages to offer their tests in the source tree. As I say in the wiki page: Other possibilities include a special .deb generated by the source (which is a bit strange and what happens to this .deb and will make it even harder to reuse upstream test suites), or putting them in the .deb to be tested (definitely wrong - most people won't want the tests and they might be very large) or having them floating about separately somewhere (which prevents us from sharing and exchanging tests with other parts of the Free Software community). The source package is always available when development is taking place. To save you looking at the wiki page for the details of the draft interface spec. I have reproduced it here: Tests/metadata ... The source package provides a test metadata file debian/tests/ control. This is a file containing zero or more RFC822-style stanzas, along these lines: Tests: fred bill bongo Restrictions: needs-root breaks-computer This means execute debian/tests/fred, debian/tests/bill, etc., each with no arguments, expecting exit status 0 and no stderr. The cwd is guaranteed to be the root of the source package which will have been built (but note that the tests must test the installed version). If the stanza contains: Tests-Directory: path-from-source-root then we execute path-from-source-root/fred, path-from-source-root/ bar, etc. This allows tests to live outside the debian/ metadata area, so that they can more palatably be shared with non-Debian distributions. Any unknown thing in Restrictions, or any unknown field in the RFC822 stanza, causes the tester core to skip the test with a message like `test environment does not support "blames-canada" restriction of test "simpsons"'. Additional possibilities: Depends: ... Tests: filenamepattern* Restrictions: modifies-global-data needs-x-display etc. - moves complexity from individual packages into central tester core. A basic test could be simply running the binary and checking the result status (or other variants of this). Eventually every package would to be changed to include at least one test. Ideally eventually where possible the upstream regression tests could be massaged so that they test the installed version. Whether this is possible and how best to achieve it has to be decided on a per-package basis. Even integration tests can be represented like this: if one package's tests Depend on the other's, then they are effectively integration tests. The actual tests can live in whichever package is most convenient. So, what do you think ? Thanks, Ian. (wearing both my Debian and Ubuntu hats) -- To UNSUBSCRIBE, email to [EMAIL PROTECTED] with a subject of "unsubscribe". Trouble? Contact [EMAIL PROTECTED]
