On Tue, 28 Nov 2017 09:33:52 +0100 Thomas Huth <th...@redhat.com> wrote:
> On 27.11.2017 23:03, John Snow wrote: > > > > On 11/23/2017 11:31 AM, Peter Maydell wrote: > [...] > >> Continuous Integration: > >> * Christian Borntraeger: qemu-iotests have broken a lot, they should be > >> run before patches are merged > > > > This, rather unfortunately, is a huge testing burden. I try to make sure > > I do it for everything I submit, but for the volume of block patches it > > really does rely CI. The more we add (to our pitifully sparse iotesting, > > I might add) the longer it takes. Ensuring per-patch testing begins to > > take prohibitively long. > > > > Perhaps per-pull or per-merge becomes more feasible. Maybe if we do > > implement a block-next amalgam we'd be able to batch our testing on a > > weekly basis. > > I think you block-layer folks should do at least run the qemu-iotests > before sending a pull request to Peter. The iotests should really not be > broken in upstream master. This is unlikely to cover the iotest failures on s390 (due to usage of ccw, strange backing devices, etc.), though. We have basically two options here: - Continue to rely on the IBM folks finding those problems (which will likely be post-merge, but better than nothing.) - Have patchew (which has a bot on s390) execute the iotests - which is time-consuming. > > >> * Peter Maydell: If it isn't tested by "make check" then it isn't tested: > >> so if something is regularly regressing then it needs to be added to > >> "make check". > > > > Is this tenable long term? We can't conceivably state that we will never > > test things that aren't in "make check" -- we ought to have different > > tiers, at least. The full testing suite should run for RC tags at least, > > but it's not feasible (I think?) to run the entire battery of tests on > > every commit... but that shouldn't stop us from running them /sometimes/... > > > > We've already got "make check SPEED=slow" for running tests that take a > lot of time. So maybe you could do that in the iotests as well, so that > the normal, quick tests can be run during "make check" and the full > iotest suite is only run during "make check SPEED=slow" ? +1 to that. Having a subset covered by default is better than nothing at all.
