Re: package testing, autopkgtest, and all that

2011-02-02 Thread Stefano Zacchiroli
On Wed, Feb 02, 2011 at 02:28:14PM +, Ian Jackson wrote: > One point I would like to make is that people who are now raising > objections to fundamental design decisions are, I think, five and a > half years too late. > > The design, both in principle and detail, was discussed in November >

Re: package testing, autopkgtest, and all that

2011-02-02 Thread Lars Wirzenius
On ke, 2011-02-02 at 17:13 +, Ian Jackson wrote: > This is easily done with autopkgtest; the only difference from your > proposal is that the source package needs to be downloaded. Doing so > is not difficult or troublesome, and can be done automatically. I concur. However, looking things fr

Re: package testing, autopkgtest, and all that

2011-02-02 Thread Michael Hanke
On Wed, Feb 02, 2011 at 05:13:21PM +, Ian Jackson wrote: > Michael Hanke writes ("Re: package testing, autopkgtest, and all that"): > > I see the point in having less by better-quality input to package > > maintainers, but again, the test results do not have to go one

Re: package testing, autopkgtest, and all that

2011-02-02 Thread Ian Jackson
Michael Hanke writes ("Re: package testing, autopkgtest, and all that"): > But to get many machines, we need to make it dead-simple to participate > in this type of croud-testing. We can have GUI frontends to let people > do specific tests, or offer "backfill" j

Re: package testing, autopkgtest, and all that

2011-02-02 Thread Michael Hanke
On Wed, Feb 02, 2011 at 04:13:37PM +, Ian Jackson wrote: > Yaroslav Halchenko writes ("Re: package testing, autopkgtest, and all that"): > > in the core -- users usually do not deal with source packages; many of > > them do not even have deb-src lines for apt. They d

Re: package testing, autopkgtest, and all that

2011-02-02 Thread Ian Jackson
Yaroslav Halchenko writes ("Re: package testing, autopkgtest, and all that"): > in the core -- users usually do not deal with source packages; many of > them do not even have deb-src lines for apt. They do not care how > things are built, but if we want them to contribu

Re: package testing, autopkgtest, and all that

2011-02-02 Thread Ian Jackson
Michael Hanke writes ("Re: package testing, autopkgtest, and all that"): > On Wed, Feb 02, 2011 at 02:28:14PM +, Ian Jackson wrote: > > As for "also" running tests which are "not part of a source package", > > it is very easy to wrap up some test

Re: package testing, autopkgtest, and all that

2011-02-02 Thread Yaroslav Halchenko
Hi Ian, thanks for the input! It does make motivation behind source packages-based testing clearer. And Simon's example is a good one ;-) As a summary: source packages-based testing often provides more convenient and upstream-friendly approach, thus it must not be "excluded", echoing Stefano's

Re: package testing, autopkgtest, and all that

2011-02-02 Thread Michael Hanke
On Wed, Feb 02, 2011 at 10:08:32AM -0500, Michael Hanke wrote: > > I don't think going back the drawing board now is a very good idea. > > What we are lacking is deployment (and, sorry for my part in the lack > > of that). > > I don't necessarily take the point of being 5 years too late. If > ever

Re: package testing, autopkgtest, and all that

2011-02-02 Thread Michael Hanke
On Wed, Feb 02, 2011 at 02:28:14PM +, Ian Jackson wrote: > As for "also" running tests which are "not part of a source package", > it is very easy to wrap up some tests in a dedicated source package if > that's desirable. The source package is then just a convenient > container format. There

Re: package testing, autopkgtest, and all that

2011-02-02 Thread Simon McVittie
On Wed, 02 Feb 2011 at 14:15:07 +, Ian Jackson wrote: > So if the tests were in binary packages, often we'd have to construct > a weird binary package which contained all or part of the built source > tree. This would be very ugly and also bulky. FWIW, Maemo does this, and it's a pain to deal

Re: package testing, autopkgtest, and all that

2011-02-02 Thread Ian Jackson
Stefano Zacchiroli writes ("Re: package testing, autopkgtest, and all that"): > All that considered, I'd like to know the rationale of this initial > design choice as well. In particular, it would be nice to know if anyone > see disadvantages in having *also* (rather the

Re: package testing, autopkgtest, and all that

2011-02-02 Thread Ian Jackson
Yaroslav Halchenko writes ("Re: package testing, autopkgtest, and all that"): > First a brief question: > > The source package provides a test metadata file > > debian/tests/control. This is a file containing zero or more > > RFC822-style stanzas, along these lines:

Re: package testing, autopkgtest, and all that

2011-02-02 Thread Michael Hanke
On Wed, Feb 02, 2011 at 08:55:07AM +0100, Stefano Zacchiroli wrote: > > So, where do we start/continue sharing the thoughts on a tentative > > DEP? ;) > > Let's see first if we have all the arguments on the table already, > thanks to this thread. I'm willing to co-drive a DEP to finalize the > spe

Re: package testing, autopkgtest, and all that

2011-02-01 Thread Stefano Zacchiroli
On Tue, Feb 01, 2011 at 06:17:45PM -0500, Yaroslav Halchenko wrote: > Unfortunately the core aspect of the current autopkgtest - relying on > tests in source packages -- imho to be not the ideal solution to > target both sides of the userbase (i.e. maintainers/QA vs mortals). Thanks for this "rela

Re: package testing, autopkgtest, and all that

2011-02-01 Thread Yaroslav Halchenko
Hi Ian, First a brief question: > The source package provides a test metadata file > debian/tests/control. This is a file containing zero or more > RFC822-style stanzas, along these lines: Do you still have somewhere that awk package demo package which had debian/tests/ ? Currently our archive doe

Re: package testing, autopkgtest, and all that

2011-01-31 Thread Ian Jackson
Lucas Nussbaum writes ("Re: package testing, autopkgtest, and all that"): > If there's a packaged tool to run the test suite on a given package, > then it's quite easy to integrate it into my infrastructure. But I > clearly do not have the time to get autopkgtest'

Re: package testing, autopkgtest, and all that

2011-01-31 Thread Lucas Nussbaum
On 31/01/11 at 00:29 +0100, Stefano Zacchiroli wrote: > The best incentive for adoption in this case is having periodic runs of > package tests, with reporting. At first glance, I'm tempted to propose > to use grid archive rebuilds to run tests. Lucas: how much work would it > be to hack your rebui

Re: package testing, autopkgtest, and all that

2011-01-31 Thread Ian Jackson
Stefano Zacchiroli writes ("Re: package testing, autopkgtest, and all that"): > > I confess that this seems to be exactly one of those cases where the > DEP process (should) shine. By putting the specification in a more > visible place than (only) in an archive packa

Re: package testing, autopkgtest, and all that

2011-01-31 Thread Ian Jackson
Stefano Zacchiroli writes ("Re: package testing, autopkgtest, and all that"): > I find "INSTALLED version of the program" to be ambiguous. It might > refer to installed in the sense of 'debian/rules install' (or, to be > more precise, 'debian/rules bin

Re: package testing, autopkgtest, and all that

2011-01-31 Thread Stefano Zacchiroli
On Thu, Jan 27, 2011 at 02:46:40PM +, Ian Jackson wrote: > > * A specification which allows a source package to declare that it > >contains tests, and how those tests need to be run. This > >specification was discussed extensively on debian-devel at the > >time and a copy is in th

Re: package testing, autopkgtest, and all that

2011-01-31 Thread Stefano Zacchiroli
On Thu, Jan 27, 2011 at 02:45:57PM +, Ian Jackson wrote: > > Ian: would you mind summarizing the status of that effort and > > comment on whether, in your opinion, people interested on this topic > > should continue from there or start over? > > Sure. autopkgtest (the codebase) isn't very big

Re: package testing, autopkgtest, and all that

2011-01-29 Thread Iustin Pop
On Fri, Jan 28, 2011 at 03:19:32PM +, Ian Jackson wrote: > Iustin Pop writes ("Re: package testing, autopkgtest, and all that"): > > I think even without a full archive integration, having this spec > > publicised a bit among developers would be useful; I know I

Re: package testing, autopkgtest, and all that

2011-01-28 Thread Ian Jackson
Iustin Pop writes ("Re: package testing, autopkgtest, and all that"): > As it seems to me, right now this is most useful for individual > maintainers to declare, and run, their own tests to ensure the built > packages are fine. A good start, I'd say. Yes, you can certain

Re: package testing, autopkgtest, and all that

2011-01-27 Thread Iustin Pop
On Thu, Jan 27, 2011 at 02:45:57PM +, Ian Jackson wrote: > Stefano Zacchiroli writes ("package testing, autopkgtest, and all that"): > > Regarding this specific point (tests run on packages as if they were > > installed), IIRC Ian Jackson worked a bit on the matter

Re: package testing, autopkgtest, and all that

2011-01-27 Thread Ian Jackson
I wrote: > * A specification which allows a source package to declare that it >contains tests, and how those tests need to be run. This >specification was discussed extensively on debian-devel at the >time and a copy is in the autopkgtest package, but I'll follow up >this email wi

Re: package testing, autopkgtest, and all that

2011-01-27 Thread Ian Jackson
Stefano Zacchiroli writes ("package testing, autopkgtest, and all that"): > Regarding this specific point (tests run on packages as if they were > installed), IIRC Ian Jackson worked a bit on the matter, producing some > code (autopkgtest---as mentioned elsewhere in

package testing, autopkgtest, and all that

2011-01-26 Thread Stefano Zacchiroli
On Mon, Jan 24, 2011 at 12:19:32AM +0100, Iustin Pop wrote: > First, tests run during a package build are good, but they do not > ensure, for example, that the package as installed is working OK. I've > been thinking that (also) providing tests to be run after the package is > installed (and not on