On 4/29/25 8:21 AM, Thomas Huth wrote:
From: Thomas Huth <th...@redhat.com>

We've got this nice vmstate-static-checker.py script that can help
to detect screw-ups in the migration states. Unfortunately, it's
currently only run manually, so there could be regressions that nobody
notices immediately. Let's run it from a functional test automatically
so that we got at least a basic coverage in each CI run.

Signed-off-by: Thomas Huth <th...@redhat.com>
---
  MAINTAINERS                      |  1 +
  tests/functional/meson.build     | 13 +++++++-
  tests/functional/test_vmstate.py | 56 ++++++++++++++++++++++++++++++++
  3 files changed, 69 insertions(+), 1 deletion(-)
  create mode 100755 tests/functional/test_vmstate.py

diff --git a/MAINTAINERS b/MAINTAINERS
index 65fb61844b3..6a8d81458ad 100644
--- a/MAINTAINERS
+++ b/MAINTAINERS
@@ -3525,6 +3525,7 @@ F: migration/
  F: scripts/vmstate-static-checker.py
  F: tests/data/vmstate-static-checker/
  F: tests/functional/test_migration.py
+F: tests/functional/test_vmstate.py
  F: tests/qtest/migration/
  F: tests/qtest/migration-*
  F: docs/devel/migration/
diff --git a/tests/functional/meson.build b/tests/functional/meson.build
index b317ad42c5a..9f339e626f6 100644
--- a/tests/functional/meson.build
+++ b/tests/functional/meson.build
@@ -76,6 +76,7 @@ tests_generic_bsduser = [
tests_aarch64_system_quick = [
    'migration',
+  'vmstate',
  ]
tests_aarch64_system_thorough = [
@@ -164,6 +165,10 @@ tests_loongarch64_system_thorough = [
    'loongarch64_virt',
  ]
+tests_m68k_system_quick = [
+  'vmstate',
+]
+
  tests_m68k_system_thorough = [
    'm68k_mcf5208evb',
    'm68k_nextcube',
@@ -230,6 +235,7 @@ tests_ppc_system_thorough = [
tests_ppc64_system_quick = [
    'migration',
+  'vmstate',
  ]
tests_ppc64_system_thorough = [
@@ -265,6 +271,10 @@ tests_rx_system_thorough = [
    'rx_gdbsim',
  ]
+tests_s390x_system_quick = [
+  'vmstate',
+]
+
  tests_s390x_system_thorough = [
    's390x_ccw_virtio',
    's390x_replay',
@@ -305,8 +315,9 @@ tests_x86_64_system_quick = [
    'migration',
    'pc_cpu_hotplug_props',
    'virtio_version',
-  'x86_cpu_model_versions',
+  'vmstate',
    'vnc',
+  'x86_cpu_model_versions',
  ]
tests_x86_64_system_thorough = [
diff --git a/tests/functional/test_vmstate.py b/tests/functional/test_vmstate.py
new file mode 100755
index 00000000000..3ba56d580db
--- /dev/null
+++ b/tests/functional/test_vmstate.py
@@ -0,0 +1,56 @@
+#!/usr/bin/env python3
+#
+# SPDX-License-Identifier: GPL-2.0-or-later
+#
+# This test runs the vmstate-static-checker script with the current QEMU
+
+import subprocess
+
+from qemu_test import QemuSystemTest
+
+
+class VmStateTest(QemuSystemTest):
+
+    def test_vmstate(self):
+        target_machine = {
+            'aarch64': 'virt-7.2',
+            'm68k': 'virt-7.2',
+            'ppc64': 'pseries-7.2',
+            's390x': 's390-ccw-virtio-7.2',
+            'x86_64': 'pc-q35-7.2',
+        }
+        self.set_machine(target_machine[self.arch])
+
+        # Run QEMU to get the current vmstate json file:
+        dst_json = self.scratch_file('dest.json')
+        self.log.info('Dumping vmstate from ' + self.qemu_bin)
+        cp = subprocess.run([self.qemu_bin, '-nodefaults',
+                             '-M', target_machine[self.arch],
+                             '-dump-vmstate', dst_json],
+                            stdout=subprocess.PIPE,
+                            stderr=subprocess.STDOUT,
+                            text=True)
+        if cp.returncode != 0:
+            self.fail('Running QEMU failed:\n' + cp.stdout)
+        if cp.stdout:
+            self.log.info('QEMU output: ' + cp.stdout)
+
+        # Check whether the old vmstate json file is still compatible:
+        src_json = self.data_file('..', 'data', 'vmstate-static-checker',
+                                  self.arch,
+                                  target_machine[self.arch] + '.json')
+        vmstate_checker = self.data_file('..', '..', 'scripts',
+                                         'vmstate-static-checker.py')
+        self.log.info('Comparing vmstate with ' + src_json)
+        cp = subprocess.run([vmstate_checker, '-s', src_json, '-d', dst_json],
+                            stdout=subprocess.PIPE,
+                            stderr=subprocess.STDOUT,
+                            text=True)
+        if cp.returncode != 0:
+            self.fail('Running vmstate-static-checker failed:\n' + cp.stdout)
+        if cp.stdout:
+            self.log.warning('vmstate-static-checker output: ' + cp.stdout)
+
+
+if __name__ == '__main__':
+    QemuSystemTest.main()

Thanks for this series Thomas, it's very useful.
Could we extend this automatically to test migration on all combinations: {qemu-system-*} x {machine}? We could generate a single list of references, containing hashes of all outputs, and a simple and clean command to regenerate all those, and associated jsons, so we don't pollute qemu code with tons of json.

This way, we can automatically detect that we never regress, not only from release to release, but commit to commit.

In case we need to update reference, people can point what's the actual difference in the commit message.

As well, since I took a look into that before, this check is not enough regarding migration. Beyonds the VMDstate, we should check as well that the default values of every field are not changed. For instance, we recently changed the default pauth property of arm cpus, and without a careful backcompat, it would have break migration. It's a bit more tricky, since there is nothing available now to dump this (I hacked that using a custom trace). And definitely not something in the scope of your series, just worth mentioning.

I hope we can one day get rid of all "Is this change safe regarding migration?" comments because we know we can trust our CI instead.

Regards,
Pierrick

Reply via email to