On September 8, 2024 1:43:33 PM UTC, Stefano Rivera <stefa...@debian.org> wrote:
>Hi Louis-Philippe (2022.11.28_01:46:58_+0200)
>> Too often, a mistake or a misconfiguration leads to no tests being detected
>> when trying to run the upstream testsuite.
>>
>> When this happens, the result of the test command typically looks like "Ran
>> 0 tests in 0.000s".
>
>A couple of years later:
>
>We've implemented the feature in Python 3.12, unittest's runner now
>exits with return value 5 if no tests were discovered, like pytest does.
>https://github.com/python/cpython/issues/113661
>
>We initially ignored this exit status in dh-python, to allow us to
>transition to Python 3.12, without too much pain.
>
>I've just done a rebuild test without ignoring it to see how much effort
>it would take to be able to use this feature:
>
>I built 6440 packages build-depending on dh-python in one way or
>another. 1483 failed, and 1124 of them say "NO TESTS RAN" in the logs.
>
>To fix these build failures, package maintainers would have these
>options:
>
>1. Get the build to run some unit tests (assuming they exist),
>2. override_dh_auto_test with something noop,
>3. export PYBUILD_DISABLE=test,
>4. We could make this failure opt-in in dh-python. Maybe via an explicit
> --test-unittest option that selects the unittest runner. If you don't
> explicitly select this runner, you'd get an attempt to run tests by
> with unittest, and no failure if no tests are found.
>
>The downside of options 2 and 3 is that if the package does start
>gaining an upstream test suite, nothing ever will attempt to run it. I
>think that's OK?
>
>If you want to experiment yourself, there's a version of dh-python in
>experimental that will treat no tests as an error.
>
>Should we file bugs for these packages? Or implement option 4?
>
>Attached is a current dd-list.
>
Thanks. I think some variation of #4 is the right answer. Causing a thousand
packages to FTBFS isn't great. I would find it easier to have an environment
variable (similar to what you suggested for #3) instead of adding an override,
but that's more of an implementation detail.
Scott K
