On Mon, 29 Jul 2024 at 12:07:27 +0000, Scott Kitterman wrote:
> While I agree that running tests is good, it's not a universally
> reasonable requirement.
I agree. In a project as large as Debian, most requirements similar to
this one at least need the qualifier "... unless there is a reason why
we can't".
> 2. There's at least one case where Debian has customizations that
> cause the test suite to fail, but the failures don't seem to cause any
> real problems. If anyone wants to make it so the weasyprint test suite
> works on Debian, please knock yourself out.
This seems like the situation is:
- a build-time test or autopkgtest test failure would be a
release-critical bug
- upstream tests failing always indicate a bug, but that bug might not
be RC:
- the tests might be broken, flaky or make assumptions that aren't
true in Debian (the tests have a bug, the package under test does not)
- or the tests might be showing us a genuine bug in the package
under test, but its severity might be anywhere between wishlist
and critical
- in the case of this specific package, Scott has assessed the severity of
the bug that is implied by the tests failing, and has decided that it's
Severity: normal or less
and that seems fine.
Ideally there would be a bug open in the BTS to document this limitation,
possibly only wishlist, possibly tagged help.
If part of the test suite is fine but a different part of the test suite
is problematic, I'd personally prefer to flag the problematic parts as
skipped/ignored, and run the rest (for example src:python3-mpd skips some
of its tests like this); but it's reasonable for opinions on this to vary,
and I don't know whether there would be anything left in this particular
package's test suite after skipping the problematic parts.
> 3. I also maintain multiple packages which require network access
> to run their test suite, so they can't run tests during build, only
> autopkgtests.
Running the test suite for these would be a Debian Policy violation
(usually a RC bug), and if Python team policy conflicts with Debian Policy,
it's Debian Policy that wins.
> If we add this as a hard requirement for team packages, what do people
> think will happen? Will people invest more volunteer time that they
> otherwise wouldn't have to add running tests? Will they leave it for
> someone else on the team to address? Will they remove packages from
> team maintenance?
These are all valid questions. It's very easy to create new QA
requirements that are simple to achieve for simple packages, but for the
packages where they are not so simple, someone who feels that a particular
level of QA coverage is valuable will need to step up to do that work,
and threatening contributors with sanctions if they don't do particular
work ("we'll throw you out of the team" or "we'll delete the package
you've invested time in from Debian" or similar) should be rare and a
last resort.
We often say that Debian Policy is (and should be) a tool for getting
a better distribution, and not a stick to beat contributors with. I
think the same should be equally true for individual teams' self-imposed
policies.
smcv
