On Tue, 20 Mar 2018 at 12:18:39 +0100, Gregor Riepl wrote: > > In case I've misunderstood you, and you're referring to unit tests > > shipped debian/tests/*, than yes, I agree. :) > > As far as I understand, these tests are executed by the package builder after > the upstream build script has finished. They're meant as a sort of integration > test, i.e. "does this package run on Debian". > > There's even a Lintian tag for them: > https://lintian.debian.org/tags/testsuite-autopkgtest-missing.html (which, I > think, is a bit overzealous)
I think you're slightly confusing build-time tests with as-installed autopkgtests, but you have the right idea. Build-time tests (dh_auto_test or similar) run the upstream "make check" or equivalent, in the same environment as the build itself (typically an autobuilder). For simple libraries, build-time tests are enough. However, for more complex packages, the autobuilder environment is too artificial or too restricted for build-time tests to give full coverage: - can't run tests as root (system-level packages like dbus often need this for good coverage) - can't contact the Internet (even if in practice you usually can, Policy says you must not, for good reasons) - can't rely on a reasonable/realistic environment, like system services running, being in a systemd-logind login session on systems that have it, or even having a home directory - can't rely on the packages being "properly installed" so that hard-coded paths can work, and have to rely on overrides like PYTHONPATH to make newly-built code visible autopkgtest (debian/tests/) is a form of as-installed testing, which takes the packages that were built, installs them in a relatively complete and realistic environment (typically a lxc container or a qemu/kvm virtual machine) and runs further tests there. Sometimes these tests just repeat the build-time test coverage, but often they can make use of the ability to do things that wouldn't work at build-time, like contacting Internet services, running things as root, or relying on system services. This often gives them better test coverage. https://wiki.gnome.org/Initiatives/GnomeGoals/InstalledTests is an example of an upstream project that is doing similar things. Because the autopkgtest container or VM is thrown away after running one package's tests in it, the tests can do things that would be unacceptable in an autobuilder environment, which again increases coverage. In Debian, autopkgtests are run separately, by different infrastructure (ci.debian.net), long after the package was built. The same package will usually be tested multiple times against newer versions of its dependencies, to look for regressions caused by a dependency change. The normal package upload/autobuild workflow does not typically run autopkgtests, although it could. Vectis, which is a personal package building tool that I'm working on, builds the package in sbuild in a virtual machine then immediately puts it through autopkgtests and piuparts testing in separate VMs, so that test failures can be checked (and either used as a reason not to upload, or ignored, based on the maintainer's judgement about the severity of the failure and the urgency of the upload - the same way the maintainer would triage Lintian errors). smcv
