Some of you have already noticed that the process which migrates uploads from the proposed pocket to the release pocket once they're ready [1] [2] is now also kicking off runs of autopkgtest [3]. I'd been holding off announcing this for a while so that we could get things working a little better, but it's really about time to explain what we're doing.
Once an upload is sufficiently ready in -proposed (that is, it has no
out-of-date binaries on amd64 and i386), proposed-migration will ask a
Jenkins instance to run autopkgtests for that package and for its
reverse-dependencies. This allows package maintainers to check not only
that their package builds and passes tests in its build tree, but that
it works properly as installed from .debs, and that it continues to work
properly when the packages it depends on change. Uploads will not be
allowed into the release pocket until all relevant tests have passed or
until any failures have been manually overridden by a member of the
release team.
Developers will be sent mail about autopkgtest failures in their
uploads. You may have received some of these already.
Some frequently asked questions:
* Why are you doing this? It makes it harder for me to land my
changes.
The whole proposed-migration process is part of a balance between
quality (only land changes that work) and velocity (make developers
more productive by letting them get changes to users more quickly).
Some of it, including this work, is loosely inspired by suggestions
from Scott James Remnant [4]. I think there's widespread agreement
that the introduction of proposed-migration has helped to make
Ubuntu's development release very much more stable and useful. There
are several classes of problems that it could never have caught in
its previous form, though, and by integrating autopkgtest I hope that
we can address that without slowing things down too much.
In the best case, an upload with associated autopkgtests will take an
extra publisher cycle to reach the release pocket. It may take
longer if its tests take longer to run. I think this is generally
acceptable, but I also think velocity is very important [5], and I'm
watching the process closely at the moment to try to minimise the
extent to which it holds anyone up.
* Why not only require tests to pass if they've passed before?
The main part of proposed-migration works on a ratchet: the result of
promoting a package must leave the number of uninstallable packages
in the archive no worse than before. For this work, though, I chose
to take a more absolutist approach: tests must pass even if they
previously failed. This is mainly because there are few enough
autopkgtests that I don't think it's an unreasonable request to fix
them; at the time of writing we have 22 failing jobs out of 166 total
[6].
* But I really need to land this change now!
Members of the ubuntu-release team can override either a failing test
for all the uploads that caused it to be run ("force-badtest") or a
single upload with some failing tests ("force-skiptest"). We hang
out on the #ubuntu-release channel on freenode, so come and explain
to us why it's more urgent to land the change than to fix the
failure.
* How do I tell what's going on with a test?
The "excuses" page has links to the Jenkins logs for each running
test. Unfortunately jobs run on a private Jenkins instance and are
only copied over to the public one after they complete, so if you
want to find out what's going on with a job in progress then you need
to ask a Canonical employee with QA lab VPN access [7] to look at the
private instance for you.
* Why do you run tests for reverse-dependencies? Won't that take ages
for core packages?
Running tests for a package itself is somewhat useful, but doesn't
gain all that much; most such tests could in principle be run as
build-time tests, although there's some benefit in running them in a
different environment, based on the installed version of the package
rather than what's in the build tree, and with only their run-time
dependencies. The real benefit comes from cross-package testing.
This means that you get to put assertions in your package that your
dependencies work, and have that tested when those dependencies
change in such a way that they only get to migrate to the release
pocket once the whole assembly is sound.
That said, we only currently run tests for direct
reverse-dependencies, not indirect ones (e.g. ubiquity -> python3 ->
python3.3), so there are some gaps in our coverage here. We may
change the details of this behaviour over time.
It is possible that there are indeed packages for which this will
take ages. If this becomes an unreasonable problem, we can force the
packages in question. At the moment, we seem to have plenty of
capacity for running tests.
* How do I reproduce test failures?
You can use the "adt-run" command in the autopkgtest package, which
supports various back-ends (for instance, I usually use "adt-run
foo.dsc --- adt-virt-schroot saucy-i386"), or you can use the more
comprehensive tools in lp:auto-package-testing. For details, see the
Ubuntu Packaging Guide [8].
Many thanks to Jean-Baptiste Lallement for doing the lion's share of the
hard work for this.
[1] https://lists.ubuntu.com/archives/ubuntu-devel/2012-October/036043.html
[2] http://people.canonical.com/~ubuntu-archive/proposed-migration/
[3] http://dep.debian.net/deps/dep8/
[4] http://netsplit.com/2011/09/08/new-ubuntu-release-process/
[5]
http://www.chiark.greenend.org.uk/ucgi/~cjwatson/blosxom/ubuntu/2011-10-24-quality-in-12-04.html
[6] https://jenkins.qa.ubuntu.com/view/Saucy/view/AutoPkgTest/
[7] https://wiki.canonical.com/UbuntuEngineering/QA/VPN
[8] http://developer.ubuntu.com/packaging/html/auto-pkg-test.html
--
Colin Watson [cjwat...@ubuntu.com]
signature.asc
Description: Digital signature
-- ubuntu-devel-announce mailing list ubuntu-devel-announce@lists.ubuntu.com https://lists.ubuntu.com/mailman/listinfo/ubuntu-devel-announce
