Peter Xu <pet...@redhat.com> writes: > On Fri, Jan 05, 2024 at 03:04:48PM -0300, Fabiano Rosas wrote: >> The migration tests have support for being passed two QEMU binaries to >> test migration compatibility. >> >> Add a CI job that builds the lastest release of QEMU and another job >> that uses that version plus an already present build of the current >> version and run the migration tests with the two, both as source and >> destination. I.e.: >> >> old QEMU (n-1) -> current QEMU (development tree) >> current QEMU (development tree) -> old QEMU (n-1) >> >> The purpose of this CI job is to ensure the code we're about to merge >> will not cause a migration compatibility problem when migrating the >> next release (which will contain that code) to/from the previous >> release. >> >> I'm leaving the jobs as manual for now because using an older QEMU in >> tests could hit bugs that were already fixed in the current >> development tree and we need to handle those case-by-case. > > Can we opt-out those broken tests using either your "since:" thing or > anything similar?
If it's something migration related, then yes. But there might be other types of breakages that have nothing to do with migration. Our tests are not resilent enough (nor they should) to detect when QEMU aborted for other reasons. Think about the -audio issue: the old QEMU would just say "there's no -audio option, abort" and that's a test failure of course. > I hope we can start to run something by default in the CI in 9.0 to cover > n-1 -> n, even if starting with a subset of tests. Is it possible? We could maybe have it enabled with "allow_failure" set. The important thing here is that we don't want to get reports of "flaky test". These tests are kind of flaky by definition, there's no way to backport a fix to the older QEMU, so there's always the chance that this test will be broken for a whole release cycle. We should act fast in adding the "since" annotation or other workaround, but that depends on our availability and the type of bug that we hit.
