Peter Xu <pet...@redhat.com> writes: > On Tue, Jan 09, 2024 at 11:46:32AM -0300, Fabiano Rosas wrote: >> Hm, it would be better to avoid the extra maintenance task at the start >> of every release, no? It also blocks us from doing n-2 even >> experimentally. > > See my other reply, on whether we can use "n-1" for migration-test. If > that can work for us, then IIUC we can avoid either "since:" or any > relevant flag, neither do we need to unmask tests after each releases. All > old tests should always "just work" with a new qemu binary.
Hmm.. There are some assumptions here: 1) New code will always be compatible with old tests. E.g. some patchseries changed code and changed a test to match the new code. Then we'd need a flag like 'since' anyway to mark that the new QEMU cannot be used with the old test. (if new QEMU is not compatible with old tests without any good reason, then that's just a regression I think) 2) There would not be issues when fixing bugs/refactoring tests. E.g. old tests had a bug that is now fixed, but since we're not using the new tests, the bug is always there until next release. This could block the entire test suite, specially with concurrency bugs which can start triggering due to changes in timing. 3) New code that can only be reached via new tests cannot cause regressions. E.g. new code is added but is kept under a machine property or migration capability. That code will only show the regression after the new test enables that cap/property. At that point it's too late because it was already released. In general I like the simplicity of your approach, but it would be annoying to change this series only to find out we still need some sort of flag later. Even worse, #3 would miss the point of this kind of testing entirely. #1 could be mitigated by a "no changes to tests rule". We'd start requiring that new tests be written and an existing test is never altered. For #2 and #3 I don't have a solution.
