Paul Gevers <elb...@debian.org> writes: > One can always file bug reports against the release.debian.org pseudo > package to ask for britney to ignore the autopkgtest result.
This would again concentrate work on a relatively small team. > One other thing that I can envision (but maybe to early to agree on or > set in stone) is that we lower the NMU criteria for fixing (or > temporarily disabling) autopkgtest in ones reverse dependencies. In > the end, personally I don't think this is up to the "relevant > maintainers" but up to the release team. And I assume that badly > maintained autopkgtest will just be a good reason to kick a package > out of testing. I already brought an example where autopkgtest ist well maintained but keeps failing. And I think that it is the package maintainers who have the experience of whether a CI test failure is critical or not. BTW, in the moment the CI tests are done in unstable -- if you want to kick out a package from *testing*, you need to test the new unstable package against this, which would be some change in the logic of autopkgtest. >> What is the reason not to use automated bug reports here? This would >> allow to use all the tools the bug system has: severities, reassigning >> closing etc. > > The largest reason is that it didn't cross my mind yet and nobody else > except you has raised the idea so far. I already don't understand this with the piuparts blocker: we have an established workflow for problems with packages that need some intervention, and this is bugs.d.o. This has a lot of very nice features, like: * discussion of the problem attached to the problem itself and stored for reference * formal documentation of problem solving in the changelog (Closes: #) * severities, tags, re-assignments, affects etc. * maintainer notifications, migration blocks, autoremovals etc. * documented manual intervention possible I don't see a feature that one would need for piuparts complaints or for CI test failures that is not in our bug system. And (I am not sure) aren't already package conflict bugs autogenerated? I would really prefer to use the bug system instead of something else. > One cravat that I see though is which system should hold the > logic. The current idea is that it is britney that determines which > combinations need to be tested and thus can use the result straight > away for the migration decision. > As Martin Pitt described in the thread I referenced in my first reply, > Ubuntu already experimented with this and they came to the conclusion > that it didn't really work if two entities have to try and keep the > logic in sync. I don't see the need to keep things in sync: If a new failure is detected, it creates an RC bug against the migration candidate, with an "affects" to the package that failed the test. The maintainer then has the possibilities: * solve the problem in his own package, upload a new revision, and close the bug there * re-assign the problem to the package that failed the test is the problem lies there. In this case, that maintainer can decide if the problem is RC, and if not, then lower the severity. In any case, the maintainers can follow the established workflow, and if one needs to look up the problems a year later, one can just search for the bug. What else would you need to keep in sync? >>> Possible autopkgtest extension: "Restrictions: unreliable"? >> >> This is not specific enough. Often you have some tests that are >> unreliable, and others that are important. Since one usually takes the >> upstream test suite (which may be huge), one has to look manually first >> to decide about the further processing. > > Than maybe change the wording: not-blocking, for-info, too-sensitive, > ignore or .... The problem is that a test suite is not that homogenious, and often one doesn't knows that ahead. For example, the summary of one of my packages (python-astropy) has almost 9000 individual tests. Some of them are critical and influence the behaviour of the whole package, but others are for a small subsystem an/or a very special case. I have no documentation of the importance of each individual test; this I decide on when I see a failure (in cooperation with upstream). But more: these 9000 tests are combined into *one* autopkgtest result. What should I put there? > If you know your test suite needs investigation, you can have it not > automatically block. But depending on the outcome of the > investigation, you can still file (RC) bugs. But then we are where we are already today: Almost all tests of my packages are "a bit complex", so I would just all mark them as non-blocking. But then I would need to file the bugs myself, and especially then there is no formal sync between the test failure and the bug. > Why I am so motivated on doing this is because I really believe this is > going to improve the quality of the release and the release process. As I already wrote: I really appreciate autopkgtest, and I would like to have a way to automatically keep and track CI test failures. I just think that it should allow the maintainer to finally overwrite, and that bugs.d.o is the superiour system for the workflow because of its flexibility. Best regards Ole
