Firstly thanks for your prompt and detailed review.
Paul Gevers writes ("Re: DEP-8 pseudo-restriction
"hint-dpkg-testsuite-triggers""):
> On 18-06-18 13:21, Ian Jackson wrote:
> > This works, in the sense that it adds the information I wanted to the
> > Testsuite-Triggers generated by dpkg-source. And it should cause all
> > DEP-8 test runners to skip the test because of the unknown
> > restriction, so it won't add any spurious test load.
>
> You are forgetting to mention the rationale for not adding this to any
> existing test. I think I know, but let's see if you have the same in mind.
Adding the dependencies explicitly to an existing test will cause
lossage (of one kind or another) if the target of the indirect
dependency changes, defeating the point of indirect package
dependencies and adding maintenance burden.
Wrong test triggers are a very minor problem compared to wrong package
dependencies, of course. And they are minor even compared to wrong
test dependencies.
> I would prefer it if we could think of a better way. We are currently
> discussing¹ if the exit codes of autopkgtest shouldn't be used further,
> e.g. by britney, and one case I may want to consider a regression is if
> an autopkgtest (without source change) went from PASS (exit code 0) to
> PASS with SKIPped tests (exit code 2). If a test has SKIPped tests by
> design that would be a shame.
I think that proposal of yours is wrong. Any package might introduce
new tests which are currently SKIP, and that is not a regression. We
don't want to discourage people from adding new, more sophisticated,
tests, even if the have requirements which can't yet be met in debci.
If you want to do this you should analyse the actual set of test
results, and check that there are no *individual tests* which went
from PASS to SKIP; just looking at the exit status is not sufficient.
The summary output from autopkgtest ought to be suitable for this.
And think this can be done without needing to teach the test
triggering infrastructure about individual tests; although you do
need to store the previous summary (whether in parsed or unparsed
form).
As a bonus, if you go in this direction, we'll end up with a more
intelligent regression blocker too: even if some set of the tests are
FAIL, regressions to the ones that are still PASS will still be
detected.
And it would be possible to mark individual tests as flaky rather than
whole packages. (I don't know how often that would be useful.)
> > Tests: hint-testsuite-triggers
> > Tests-Directory: tests/tests
>
> Tests-Directory is cruft if you don't expect the test to be run anyways.
Yes, but it's generated by my template, and is harmless, so I haven't
bothered to add code to suppress it. (Indeed in my actual package,
because of the way I handle test enumeration, the test file exists and
would succeed if you ran it...)
> > I chose to invent the restriction name `hint-dpkg-testsuite-triggers'
>
> ^^^^^ dropped below.
Err, yes, I changed my mind about the name while writing the email.
> > The packages listed as Depends for this test are usually indirect
> > dependencies, updates to which are considered likely to cause
> > regressions in other tests defined in this package.
>
> I don't like the word "likely" here. Have some trust in your peers. At
> most this is "elevated risk". Or "have in the past led to regressions"
If you look below at my specific use cases, you can see that the
changes to diff/patch which might cause regressions are only likely to
do so due to infelicity in dpkg-dev, an intervening package. Quite
reasonable improvements in diff/patch might break things, and require
remedial work (ideally, then, in dpkg-dev).
So I certainly don't want to imply any insults towards the named
packages. They are often likely to be ones where the interactions are
indirect but still intimate.
How about "more likely" ? Or "are considered to pose a risk of
regressions" ?
> > There is currently no way to specify this hint on a per-test
> > basis; but in any case the debian.org machinery is not able to
> > think about triggering individual tests.
>
> Are you considering the case here where we would trigger one test out of
> the autopkgtest for pkg-a and another for pkg-b? Before that is
> possible, we would need to change the Testsuite-Triggers mechanism again.
Yes. Or the system might discover that there is no point triggering
any tests because all of the relevant ones have unmeetable
restrictions, and all of the runnable ones are irrelevant to the
update being performed.
The Testsuite-Triggers mechanism is indeed rather crude. I don't like
it very much but of course we didn't want the best to be the enemy of
the good.
What I'm proposing here is indeed awkward. It is conceptually wrong -
a fact which is highlighted in my description - but that conceptual
wrongness is purely inherited from the Testsuite-Triggers
arrangements, which I am trying to manipulate.
If there are better alternatives that are likely to be deployable soon
then I'm all ears...
> > I haven't yet pushed the change with this restriction name in it,
> > anywhere, so if people think a different name is better I'm open to
> > suggestions.
>
> I don't really care what name this carries though if we go for it.
OK.
> > I wrote:
> >> 4. Can we have a way to trigger tests from updates of non-direct
> >> rdepends ? At some point in the future maybe we will run tests of
> >> whole batches of updates and then have some algorithm to chop out
> >> what the failures are caused by, but for now it would be useful to
> >> be able to declare a specific indirect dependency for test trigger.
> >> Maybe an XS- header field ?
> >
> > In dgit, I wanted to do this because ISTM that some of the most likely
> > sources of regressions are a handful of indirect dependencies.
>
> Just curious, but if they are so important for your package, how come
> they end up being only indirect dependencies?
Happy to answer that:
* gnupg:
gnupg2 has a bunch of daemons and stuff which it tries to
autostart, but the autostart and clean up is plagued with races and
other bugs. On a number of previous occasions the dgit test suite
has started failing because the gnupg races got worse, making my
workarounds (which are gradually becoming more horrific)
insufficient.
I don't want to add a direct dependency because dgit is very
agnostic about what `gpg' on PATH is actually like, and the package
name `gnupg' means different things in different Debian releases.
Because dgit Depends on devscripts and does much of its signing
with debsign, there is guaranteed to be something suitable.
Certainly I don't want to stop anyone installing gnupg1 if they can
do so, nor do I want to stop them running the test suite with
gnupg1.
(Maybe this entry in Testsuite-Triggers is of no practical import
with the current test runner, because dgit-infrastructure Depends
on gpgv, so gnupg2 updates will cause dgit's tests to be rerun
anyway.)
* patch/diffutils:
dgit is quite fussy about how source packages with patch queues
work. dpkg-source is of course responsible for the actual
implementation, which is in terms of diff and patch. dpkg-source
is *not* careful about exactly what the semantics of diff and patch
are.
We have already had one incident where a change to the behaviour of
diff and patch led to an uncontrolled and unversioned semantic
change to the feature set of the `3.0 (quilt)' source package
format. (As a result, some newer packages cannot be unpacked on
older Debian releases, and this was not detected until far too
late.)
Rerunning the dgit autopkgtests when diff or patch is updated seems
like a sensible precaution to me. (Better would be for dpkg-source
to have a comprehensive DEP-8 test suite would detect accidentally
picking up new diff/patch features, and which avoids the meaning of
existing source packages changing.)
HTH.
Ian.
