Hello Steve, On Sat, Jan 14, 2017 at 10:15:15AM -0800, Steve Langasek wrote: > If the failure of the test is not critical, then it should not be used > as a gate for CI. Which means you, as the package maintainer who > knows that this test failure is not critical, should fix your > autopkgtest to not fail when the non-critical test case fails.
If we make it so that the only way to mark a test failure as non-critical is to hack the test suite to exit zero anyway, we would make it much less convenient to run non-critical tests on ci.debian.net. Maintainers could no longer look for 'fail' to see whether their non-critical tests have failed: they would have to open up the test output. I agree with the principle that test failures should be RC by default. I think we need an additional field in d/tests/control to mark individual tests as non-critical (this wouldn't really help Ole's 9000 tests package though). -- Sean Whitton
signature.asc
Description: PGP signature
