Rationale
---------
Today we have a few test cases in the QEMU tree that just run
QEMU with a few command-line options, run some QMP or HMP
commands, and check if QEMU didn't crash.
Some examples:
* scripts/device-crash-test
* The test case suggested by Thomas at:
Subject: [RFC PATCH] tests: Add a device_add/del HMP test
* Some test cases in tests/device-introspect-test.c
* tests/cpu-plug-test.c
* tests/display-vga-test.c
* tests/drive_del-test.c
* tests/machine-none-test.c
* tests/test-hmp.c
* Some device test skeletons: ac97-test.c, e1000-test.c,
eepro100-test.c, es1370-test.c, i82801b11-test.c,
intel-hda-test.c, ioh3420-test.c, ipoctal232-test.c,
ne2000-test.c, nvme-test.c, pcnet-test.c, spapr-phb-test.c,
test-netfilter.c, tpci200-test.c, usb-hcd-ohci-test.c,
usb-hcd-xhci-test.c, virtio-balloon-test.c,
virtio-console-test.c, virtio-serial-test.c, vmxnet3-test.c.
All of those test cases require lots of boilerplate code just to
run some combinations of QEMU command-line options and monitor
commands.
In addition to that, writing and reviewing test code like the
examples above is more complex than it could be. Some past
examples from qemu-devel:
* Subject: [RFC 5/6] device-crash-test: Basic device_add support
* Subject: [RFC 6/6] device-crash-test: Multi-device device_add test
* Subject: [RFC PATCH] tests/device-introspect: Test devices with all machines,
not only with "none"
The Proposal
------------
This series introduces the scripts/validator.py test runner, and
a set of example test specifications in tests/validator.
Each test specification is a YAML file containing the QEMU
command-line and monitor commands to run, and can contain
variables like $MACHINE, $DEVICE, $CPU, that can be expanded
automatically by the test runner.
This way, scripts/device-crash-test can be replaced by a one-line
test specification:
command-line: '$QEMU -S -machine $MACHINE,accel=$ACCEL -device $DEVICE'
The device_add/device_del test case written by Thomas can be specified as:
command-line: '$QEMU -S -machine $MACHINE,accel=$ACCEL'
monitor-commands:
- qmp:
- execute: 'device_add'
arguments:
driver: '$DEVICE'
id: 'test-device-for-$DEVICE'
- execute: 'device_del'
arguments:
id: 'test-device-for-$DEVICE'
and the test runner will take care of running test cases with
with all valid machine-types, device-types, and accelerators.
For other examples, see the last patch in this series
("Collection of validator.py test cases"). Most of the examples
replace existing C or Python test cases with simple declarative
test specifications.
I expect this to be very useful when somebody is fixing a QEMU
crash and just want to add a very simple reproducer to ensure the
crash will never be introduced again.
(I actually suggest we _require_ people to include a
tests/validator test specifications every time they fix a crash
that can be reproduced using just QMP commands).
Features
--------
Current features:
* Automatic variable expansion, as described above
* A simple method to specify defaults (e.g. to test only
"-machine none" or only "accel=kvm:tcg" unless running in --full mode)
* A simple method to represent expected failures (it can't
replace the device-crash-test whitelist mechanism, yet)
Caveats
-------
The test runner code still needs some clean up. I'm not happy with the current
variable enumeration/expansion code, and plan to make it cleaner.
How to test
-----------
You can run all example test cases on /usr/bin/qemu-system-x86_64 with:
$ ./scripts/validator.py tests/validator/* -V QEMU=/usr/bin/qemu-system-x86_64
However, the device-crash-test.yaml test specification still
takes too long to run and reports too many false positives. So I
suggest doing this instead:
$ ./scripts/validator.py $(ls tests/validator/* | grep -v device-crash-test) \
-V QEMU=/usr/bin/qemu-system-x86_64
On my system, this runs ~700 test cases in ~15 seconds.
What's out of scope?
--------------------
The following are _not_ goals of this system:
* Replacing the work done by the Avocado team to support
Avocado-based tests.
* Making the YAML test specification a fully-featured imperative
language. Anything that requires extra logic still needs to be
written in C or Python.
Future Plans
------------
Short term:
* Choose a better name for the test runner
* Support a list of known-failures similar to the whitelist
in device-crash-test.
* Replace device-crash-test completely
* "make check" rules
Long term:
* stdout and QMP command output validation.
I plan to use this to write machine-type/guest-ABI
compatibility test cases that will detect guest ABI breakage as
soon as possible.
* See if the Avocado multiplexer API can be used to implement
variable expansion.
* See if we can make some of the features (e.g. automatic
variable expansion) available to Python test cases
* Fuzzing QMP commands and QEMU command-line arguments.
* See if code can be shared with the iotests test
runner (especially the variable expansion logic)
Eduardo Habkost (18):
qmp.py: Make it safe to call close() any time
qmp.py: Fix error handling for Python 3
qmp.py: Cleanly handle unexpectedly closed socket
qemu.py: Make _vm_monitor a method
qemu.py: Split _base_args()
qemu.py: Move _load_io_log() call to _post_shutdown()
qemu.py: Use wait() logic inside shutdown()
qemu.py: Close _qmp inside _post_shutdown()
qemu.py: Make monitor optional
qemu.py: Set _launched = False on _post_shutdown
qemu.py: Log crashes inside _post_shutdown()
qemu.py: Only wait for process if it's still running
qemu.py: 'force' parameter on shutdown()
qemu.py: Don't try to quit cleanly on exceptions
qemu.py: qmp_obj() method
qemu.py: is_launched() method
validator.py script
Collection of validator.py test cases
scripts/qemu.py | 112 +++--
scripts/qmp/qmp.py | 16 +-
scripts/validator.py | 724 +++++++++++++++++++++++++++++++++
tests/validator/cpu.yaml | 5 +
tests/validator/device-add.yaml | 17 +
tests/validator/device-crash-test.yaml | 7 +
tests/validator/device-help.yaml | 3 +
tests/validator/device-introspect.yaml | 23 ++
tests/validator/device-list.yaml | 10 +
tests/validator/just-help.yaml | 3 +
10 files changed, 876 insertions(+), 44 deletions(-)
create mode 100755 scripts/validator.py
create mode 100644 tests/validator/cpu.yaml
create mode 100644 tests/validator/device-add.yaml
create mode 100644 tests/validator/device-crash-test.yaml
create mode 100644 tests/validator/device-help.yaml
create mode 100644 tests/validator/device-introspect.yaml
create mode 100644 tests/validator/device-list.yaml
create mode 100644 tests/validator/just-help.yaml
--
2.14.3
