On Mon, 21 Sep 2020 at 09:09:51 -0300, Antonio Terceiro wrote: > Maybe you could include something like this (the wording can be improved): > > Note, however, that such superficial tests are still somewhat useful, > as they will be considered, for example, to block dependencies from > breaking your package. In other words, please do not react to this bug > report by dropping tests from your package completely. More extensive > testing is of course better, but even superficial tests are better for > the overal quality of Debian than no tests at all.
Perhaps a more positive way to phrase the bug report would be to lead with this, and then go on to say why these tests should be marked superficial? That will hopefully reduce the tendency for maintainers to remove tests in response. It's also really useful for the template that is discussed on -devel to mention the usertags that are going to be used (if any), so that it's easy to search for them in a machine-readable way. It looks as though the bugs that have already been filed are usertagged to appear in <https://udd.debian.org/cgi-bin/bts-usertags.cgi?user=sudipm.mukherjee%40gmail.com&tag=superficialtest> so let's go with that. In cases like this where the change is so trivial to make and so similar across multiple packages, it's probably also useful to link to examples of a maintainer making this change correctly, like [3] and [4] below. Maybe something like this (I'm making some assumptions here about what the release team does and doesn't encourage, so don't actually use this wording until someone from the RT has acknowledged it): ----8<---- Subject: <package>: autopkgtest must be marked superficial Severity: important User: sudipm.mukher...@gmail.com Usertags: superficialtest It has been noticed that the autopkgtest in <package> is running a trivial command that does not provide significant test coverage: - <command being run, e.g. foo --help> These superficial tests are a useful way to detect regressions in dependencies and prevent them from breaking your package, and the Release Team encourages their inclusion in packages that do not have a more thorough test suite available. However, it is important that we are realistic about the level of test coverage provided by these commands: most regressions cannot be detected in this way, so it is not appropriate for packages that only have superficial tests to migrate to the testing distribution with less opportunity for testing by users than if these tests had not been present. To support this, the keyword "Restrictions: superficial" has been defined [1]. Packages where all tests are marked with this keyword are not considered for the reduced migration age from unstable to testing, and will not be allowed to migrate automatically in later stages of the freeze [2]. Please mark all superficial autopkgtests with this keyword, similar to [3] or [4]. The Release Team has listed this issue in the list of Release Critical Issues for bullseye [5] and has mentioned that the test must be marked superficial if it is not testing one of its own installed binary packages in some way. As a result, the severity of this bug report might be increased to serious in future. [1] https://salsa.debian.org/ci-team/autopkgtest/-/blob/master/doc/README.package-tests.rst#defined-restrictions [2] https://release.debian.org/bullseye/freeze_policy.html [3] https://salsa.debian.org/utopia-team/dbus/-/commit/a80908df7d119b181eec5eb0542634a30c2ad468 [4] https://salsa.debian.org/apparmor-team/apparmor/-/commit/580667513a097088ebe579884b38ac8d8666d3b3 [5] https://release.debian.org/bullseye/rc_policy.txt ----8<---- Regards, smcv
