On Thu, Mar 14, 2019 at 03:57:06PM +0000, Alex Bennée wrote: > > Hi, > > As we approach stabilisation for 4.0 I thought it would be worth doing a > review of the current state of CI and stimulate some discussion of where > it is working for us and what could be improved. > > Testing in Build System > ======================= > > Things seem to be progressing well in this respect. More and more tests > have been added into the main source tree and they are only a make > invocation away. These include: > > check (includes unit, qapi-schema, qtest and decodetree) > check-tcg (now with system mode tests!) > check-softfloat > check-block > check-acceptance > > Personally check-acceptance is the area I've looked at the least but > this seems to be the best place for "full life cycle" tests like booting > kernels and running stress and performance tests. I'm still a little > unsure how we deal with prebuilt kernels and images here though. Are > they basically provided by 3rd parties from their websites? Do we mirror > any of the artefacts we use for these tests?
While it's possible to add any sort of files alongside the tests, and "get it"[1] from the test[2], this is certainly not desirable for kernels and other similarly large blobs. The current approach is to use well known URLs[3] and download[4][5] those at test run time. Those are cached locally, automatically on the first run and reused on subsequent executions. The caching is helpful for development environments, but is usually irrelevant to CI environments, where you'd most often than not get a new machine (or a clean environment). For now I would, also for the sake of simplicity, keep relying on 3rd party websites until they prove to be unreliable. This adds trasnparency and reproducibility well beyond can be achieved if we attempt to mirror them to a QEMU sponsored/official location IMO. > > One area of concern is how well this all sits with KVM (and other HW > accelerators) and how that gets tested. With my ARM hat on I don't > really see any integration between testing kernel and QEMU changes > together to catch any problems as the core OS support for KVM gets > updated. > In short, I don't think there should be at the QEMU CI be any integration testing that changes both KVM and QEMU at once. But, that's me assuming that the vast majority of changes in QEMU and KVM can be developed, an tested separately of each other. That's in sharp contrast with the the days in which KVM Autotest would build both the kernel and userspace as part of all test jobs, because of very frequent dependencies among them. I'd love to get feedback on this from KVM (and other HW accelerator) folks. > Another area I would like to improve is how we expand testing with > existing test suites. I'm thinking things like LTP and kvm-unit-tests > which can exercise a bunch of QEMU code but are maybe a bit to big to be > included in the source tree. Although given we included TestFloat (via a > git submodule) maybe we shouldn't dismiss that approach? Or is this > something that could be done via Avocado? > Well, there's this: https://github.com/avocado-framework-tests/avocado-misc-tests Which contains close to 300 tests, most of them wrappers for other test suites, including LTP: https://github.com/avocado-framework-tests/avocado-misc-tests/blob/master/generic/ltp.py I'm claiming it's the perfect fit for your idea, but sounds like a good starting point. > Generally though I think we are doing pretty well at increasing our test > coverage while making the tests more directly available to developers > without having to rely on someones personal collection of random > binaries. > +1. > I wanted to know if we should encode this somewhere in our developer > documentation: > > There is a strong preference for new QEMU tests to be integrated with > the build system. Developers should be able to (build and) run the new > tests locally directly from make. > > ? > There should definitely be, if reasonable, a similar experience for running different types of tests. Right now, the build system (make targets) is clearly the common place, so +1. - Cleber. [1] - https://avocado-framework.readthedocs.io/en/69.0/api/core/avocado.core.html#avocado.core.test.TestData.get_data [2] - https://avocado-framework.readthedocs.io/en/69.0/WritingTests.html#accessing-test-data-files [3] - https://github.com/clebergnu/qemu/blob/sent/target_arch_v5/tests/acceptance/boot_linux_console.py#L68 [4] - https://github.com/clebergnu/qemu/blob/sent/target_arch_v5/tests/acceptance/boot_linux_console.py#L92 [5] - https://avocado-framework.readthedocs.io/en/69.0/WritingTests.html#fetching-asset-files
