Re: pybuild-autopkgtest (was: Notes from the DC22 Python Team BoF)

[Louis-Philippe Véronneau]

On 2022-07-27 16 h 32, Julian Gilbey wrote:

On Wed, Jul 27, 2022 at 07:45:12PM +0100, Julian Gilbey wrote:

On Wed, Jul 27, 2022 at 10:26:33AM -0400, Louis-Philippe Véronneau wrote:

The way I see it:

1. We should have a Lintian tag for packages not using the new
pybuild-autodep8 autopkgtest. It would be even better if this tag would be a
pointed hint that identified 'manually' written unit test autopkgtests that
could be replaced.

This way, you get something like:

python-foo source: not-using-pybuil-autodep8 [debian/tests/unittests]

for python packages that have old 'manually' written unit test autopkgtests
and:

python-foo source: not-using-pybuild-autodep8 [no-autopkgtest]

for python packages without any autopkgtest.

2. lintian-brush (or something else, but I think lintian-brush is the right
tool) would go over these packages to:

2.1 Add the new autodep8 autopkgtests and build the package to see if they
pass
2.2 Remove the "manual" unit test autopkgtests if 2.1 succeeds
2.3 Open a bug report if 2.1 fails

I'd be wary about 2.2 and 2.3.  I have several packages where I know
that an automated test will fail; there are all sorts of weird cases
where I've had to write tests manually.  I would also be quite cross
if manually crafted tests were automatically removed, especially in
cases such as Simon mentioned where they do things that that
automatically generated test does not do.  Another thing I could
imagine happening is that the automated test succeeds in a trivial way
- it succeeds but doesn't actually test much because of the nature of
the package.

On the other hand, a bug report saying something like the following
seems much more reasonable: "We've tested this package using the
automated autopkgtest system and it seems to work by adding the line
'Testsuite: autopkgtest-pkg-pybuild'; please check that the automated
tests cover all of the tests of your manually written debian/tests/*
and if so, then please remove them.  The autopkgtest-pkg-pybuild logs
are attached."  This would give the maintainer the chance to decide
how best to proceed.

Here's another alternative to steps 2.1-2.3 based on this:

  For packages which currently have manually-written autopkgtests:

  2.A Try removing debian/tests and adding Testsuite:
      autopkgtest-pkg-pybuild to debian/control, then building the
      package and running autopkgtest.

  2.B If this works, then submit a bug report to the BTS as I suggested
      above.

  2.C If this does not work, don't do anything more; trust that the
      maintainer knew what they were doing when they wrote the manual
      autopkgtests.

  For packages which don't currently have manually-written
  autopkgtests:

  2.A' Try adding Testsuite: autopkgtest-pkg-pybuild to debian/control,
       then building the package and running autopkgtest.

  2.B' If this works, then either Janitor adds this line to
       debian/control or submit a bug report to the BTS to recommend
       this.  (But we would not expect Janitor to do step 2.1', so this
       might have to be a different setup, or maybe the script doing
       2.A' could leave a list of packages for Janitor, or something
       like that.)

  2.C' If this does not work, submit a wishlist bug to the BTS to
       recommend that the maintainer adds some autopkgtest tests,
       either by making the autodep8 system work or by writing some
       manual tests.

I'd be wary about adding lintian tags for this, though: with so many
packages not being able to use the autodep8 system (I vaguely recall
someone suggesting that a third of Python packages would not be able
to use the system), that's a lot of packages getting false positives.
In particular, for the suggested first version of
not-using-pybuild-autodep8 tag (which would probably be better named
manual-autopkgtest-could-be-pybuild-autodep8), how would lintian go
about identifying which packages fall into category 2.B and which into
2.C?  The second version of the tag (better named something like
no-autopkgtest-could-use-pybuild-autodep8, but that's still not very
good) is less problematic.

--
  ⢀⣴⠾⠻⢶⣦⠀
  ⣾⠁⢠⠒⠀⣿⡁  Louis-Philippe Véronneau
  ⢿⡄⠘⠷⠚⠋   po...@debian.org / veronneau.org
  ⠈⠳⣄

This all makes a lot of sense to me and I think this is the way forward, once the MR is merged. Thanks for the discussion.

OpenPGP_0xE1E5457C8BAD4113.asc
Description: OpenPGP public key

OpenPGP_signature
Description: OpenPGP digital signature