Cleber Rosa <cr...@redhat.com> writes:
> On Fri, Dec 06, 2019 at 09:00:10AM -0500, Wainer dos Santos Moschetta wrote: >> This series add a new acceptance test: boot an uncompressed >> Linux kernel built with CONFIG_PVH, so checking the PVH >> capability introduced in QEMU 4.0 works. >> >> The test implementation depends on [1] which is likely released >> on next Avocado. So that will need a version 2 of this >> series to bump Avocado version. >> > > Right, the Avocado bits have been merged, so unless a major reversal > comes (not expected at all), it will be on Avocado 74.0. > >> Also I want to use this as an example of a scenario that test >> assets could be better managed. As you see on patch 01 the >> kernel is built at test time on localhost. While Avocado provides >> an API to easily fetch and build it, the whole process takes >> reasonable time - besides the fact that localhost must have >> all build dependencies installed. How could it be done better? >> > > This is clearly not a "kernel build" test, so we should avoid building > the kernel as part of the "PVH boot" test. The problem you expose > here is a very real, and each possible solution has its own problems, > unfortunately. > > The best solution IMO would be to find a "well known" distribution of > such a kernel. Maybe something maintained by the Xen project or one > of its commercial products? > > The second best solution is to have a helper script (using the Avocado > utils API is fine) that will build a kernel and create/populate the > test's data directory (tests/acceptance/pvh.py.data/) with a vmlinux. > The test can cancel itself if it doesn't find a kernel there. > > The third best solution IMO is for this test to require a parameter > telling where the CONFIG_PVH enabled vmlinux kernel image is. > > Also notice that, with a custom built kernel image (a different one > for each user), the result of the test as a general indicator to the > QEMU codebase diminishes quite a bit. I can see a use case for making it easier for developers to build test cases otherwise everyone has their own particular variant of the kernel and buildroot/busybox which they have in their own farm. However as Cleber has noted they make poor standardised tests given the variation in kernel builds you can get. That said I think there are better targets. kvm-unit-tests can be built against a range of architectures and are fashioned as individual unit tests for specific functionality. It would be useful to make it easy for a developer to regenerate the test assets to re-run a test someone else has found to fail. >> Nonetheless I argue in favor of merging this as is, and >> gradually implement corrections to improve the tests assets >> management. Also if eventually the test is proven to unacceptable >> slow down the Travis CI then I can add a guard to skip it. >> > > I'm pretty sure that building the kernel will cause a major slow down > of the Travis CI results for the "acceptance" block (and thus for the > overall results). But, it may also time it out (there's a 50 minutes > deadline). > > I hope this brings some ideas, and thanks for coming up with > interesting problems! :) > > - Cleber. > >> [1] https://github.com/avocado-framework/avocado/pull/3389 >> >> Wainer dos Santos Moschetta (2): >> tests/acceptance: Add PVH boot test >> .travis.yml: Add kernel build deps for acceptance tests >> >> .travis.yml | 2 ++ >> tests/acceptance/pvh.py | 48 +++++++++++++++++++++++++++++++++++++++++ >> 2 files changed, 50 insertions(+) >> create mode 100644 tests/acceptance/pvh.py >> >> -- >> 2.21.0 >> -- Alex Bennée
