Re: Bug#1024971: pybuild: should fail when the result of running tests is "Ran 0 tests in 0.000s"
Hi Louis-Philippe (2022.11.28_01:46:58_+0200) > Too often, a mistake or a misconfiguration leads to no tests being detected > when trying to run the upstream testsuite. > > When this happens, the result of the test command typically looks like "Ran > 0 tests in 0.000s". A couple of years later: We've implemented the feature in Python 3.12, unittest's runner now exits with return value 5 if no tests were discovered, like pytest does. https://github.com/python/cpython/issues/113661 We initially ignored this exit status in dh-python, to allow us to transition to Python 3.12, without too much pain. I've just done a rebuild test without ignoring it to see how much effort it would take to be able to use this feature: I built 6440 packages build-depending on dh-python in one way or another. 1483 failed, and 1124 of them say "NO TESTS RAN" in the logs. To fix these build failures, package maintainers would have these options: 1. Get the build to run some unit tests (assuming they exist), 2. override_dh_auto_test with something noop, 3. export PYBUILD_DISABLE=test, 4. We could make this failure opt-in in dh-python. Maybe via an explicit --test-unittest option that selects the unittest runner. If you don't explicitly select this runner, you'd get an attempt to run tests by with unittest, and no failure if no tests are found. The downside of options 2 and 3 is that if the package does start gaining an upstream test suite, nothing ever will attempt to run it. I think that's OK? If you want to experiment yourself, there's a version of dh-python in experimental that will treat no tests as an error. Should we file bugs for these packages? Or implement option 4? Attached is a current dd-list. Stefano -- Stefano Rivera http://tumbleweed.org.za/ +1 415 683 3272 dh-python-no-unittests.packages.txt.xz Description: application/xz dh-python-no-unittests.ddlist.txt.xz Description: application/xz
Re: Bug#1024971: pybuild: should fail when the result of running tests is "Ran 0 tests in 0.000s"
On September 8, 2024 1:43:33 PM UTC, Stefano Rivera wrote: >Hi Louis-Philippe (2022.11.28_01:46:58_+0200) >> Too often, a mistake or a misconfiguration leads to no tests being detected >> when trying to run the upstream testsuite. >> >> When this happens, the result of the test command typically looks like "Ran >> 0 tests in 0.000s". > >A couple of years later: > >We've implemented the feature in Python 3.12, unittest's runner now >exits with return value 5 if no tests were discovered, like pytest does. >https://github.com/python/cpython/issues/113661 > >We initially ignored this exit status in dh-python, to allow us to >transition to Python 3.12, without too much pain. > >I've just done a rebuild test without ignoring it to see how much effort >it would take to be able to use this feature: > >I built 6440 packages build-depending on dh-python in one way or >another. 1483 failed, and 1124 of them say "NO TESTS RAN" in the logs. > >To fix these build failures, package maintainers would have these >options: > >1. Get the build to run some unit tests (assuming they exist), >2. override_dh_auto_test with something noop, >3. export PYBUILD_DISABLE=test, >4. We could make this failure opt-in in dh-python. Maybe via an explicit > --test-unittest option that selects the unittest runner. If you don't > explicitly select this runner, you'd get an attempt to run tests by > with unittest, and no failure if no tests are found. > >The downside of options 2 and 3 is that if the package does start >gaining an upstream test suite, nothing ever will attempt to run it. I >think that's OK? > >If you want to experiment yourself, there's a version of dh-python in >experimental that will treat no tests as an error. > >Should we file bugs for these packages? Or implement option 4? > >Attached is a current dd-list. > Thanks. I think some variation of #4 is the right answer. Causing a thousand packages to FTBFS isn't great. I would find it easier to have an environment variable (similar to what you suggested for #3) instead of adding an override, but that's more of an implementation detail. Scott K
Re: Bug#1024971: pybuild: should fail when the result of running tests is "Ran 0 tests in 0.000s"
On Sun, Sep 08, 2024 at 03:43:33PM +0200, Stefano Rivera wrote: > [...] > We've implemented the feature in Python 3.12, unittest's runner now > exits with return value 5 if no tests were discovered, like pytest does. > https://github.com/python/cpython/issues/113661 > [...] > > I built 6440 packages build-depending on dh-python in one way or > another. 1483 failed, and 1124 of them say "NO TESTS RAN" in the logs. My guess is that most of these 1124 have no tests at all, rather than having a misconfigured setup. A unittest is the pybuild default test framework, unittest is used and fails to find any tests, hence all of these failures. > To fix these build failures, package maintainers would have these > options: > > 1. Get the build to run some unit tests (assuming they exist), > 2. override_dh_auto_test with something noop, > 3. export PYBUILD_DISABLE=test, > 4. We could make this failure opt-in in dh-python. Maybe via an explicit >--test-unittest option that selects the unittest runner. If you don't >explicitly select this runner, you'd get an attempt to run tests by >with unittest, and no failure if no tests are found. I like option 4 for the above reason. But implementing this would mean that all of the packages that currently *do* use unittest (intentionally, but without having to code it explicitly as it's the default) would suddenly not have any tests running until they proactively add --test-unittest or set PYBUILD_TEST_UNITTEST = 1 or similar. This seems like an unfortunate consequence. Is there a way of looking at the logs of the packages that passed the build to identify which ones successfully passed tests using unittest? There is also the issue of packages that use unittest (as the default) via autopkgtest-pkg-pybuild but override dh_auto_test during the build, though that will be much rarer. Best wishes, Julian
