Re: [PATCH 1/3] Acceptance tests: bump pycdlib version for easier installation

[Philippe Mathieu-Daudé]

On 10/9/20 10:55 PM, Cleber Rosa wrote:

On with certain versions of "pip", package installations will attempt
to create wheels.  And, on environments without a "complete" Python
installation (as described in the acceptance tests requirements docs),
that will fail.

pycdlib, starting with version 1.11.0, is now being made available
as wheels, so its instalation on those constrained environments is
now possible.

Cc: Bug 1897783 <1897...@bugs.launchpad.net>
Buglink: https://bugs.launchpad.net/qemu/+bug/1880189


This BZ is different. The correct URL is:
https://bugs.launchpad.net/qemu/+bug/1897783

Reviewed-by: Philippe Mathieu-Daudé 
Tested-by: Philippe Mathieu-Daudé 


Reported-by: Philippe Mathieu-Daudé 
Signed-off-by: Cleber Rosa 
---
  tests/requirements.txt | 2 +-
  1 file changed, 1 insertion(+), 1 deletion(-)

diff --git a/tests/requirements.txt b/tests/requirements.txt
index 036691c922..a1c631fa59 100644
--- a/tests/requirements.txt
+++ b/tests/requirements.txt
@@ -2,4 +2,4 @@
  # in the tests/venv Python virtual environment. For more info,
  # refer to: https://pip.pypa.io/en/stable/user_guide/#id1
  avocado-framework==81.0
-pycdlib==1.9.0
+pycdlib==1.11.0

Re: [PATCH v2] tests/acceptance: Test case for detecting -object crashes

[Philippe Mathieu-Daudé]

On 10/9/20 10:29 PM, Eduardo Habkost wrote:

Add a simple test case that will run QEMU directly (without QMP)
just to check for crashes when using `-object`.

Signed-off-by: Eduardo Habkost 
---
Changes v1 -> v2:
* "Running command:" log message instead of "Command:" (Cleber)
* Use universal_newlines=True instead of encoding='utf-8' (Cleber)
* Rename devices() to get_devices() (Cleber)
* Use @avocado.fail_on(subprocess.CalledProcessError) (Cleber)
* Reword test_crash() docstring (Cleber)
* Reorder imports


Assuming:
Based-on: <20201008202713.1416823-1-ehabk...@redhat.com>

I get:

 (1/2) tests/acceptance/object_option.py:ObjectOption.test_help: 
qemu-system-avr: No machine specified, and there is no default

Use -machine help to list supported machines
FAIL: CalledProcessError(1, ['./qemu-system-avr', '-object', 'help']) 
(0.19 s)
 (2/2) tests/acceptance/object_option.py:ObjectOption.test_crash: 
qemu-system-avr: No machine specified, and there is no default

Use -machine help to list supported machines
FAIL: CalledProcessError(1, ['./qemu-system-avr', '-object', 'help']) 
(0.18 s)



---
  tests/acceptance/object_option.py | 53 +++
  1 file changed, 53 insertions(+)
  create mode 100644 tests/acceptance/object_option.py

diff --git a/tests/acceptance/object_option.py 
b/tests/acceptance/object_option.py
new file mode 100644
index 00..511c03a36f
--- /dev/null
+++ b/tests/acceptance/object_option.py
@@ -0,0 +1,53 @@
+# Copyright (c) 2020 Red Hat, Inc.
+#
+# Author:
+#  Eduardo Habkost 
+#
+# This work is licensed under the terms of the GNU GPL, version 2 or
+# later.  See the COPYING file in the top-level directory.
+import shlex
+import subprocess
+
+import avocado
+import avocado_qemu
+
+
+class ObjectOption(avocado_qemu.Test):
+"""Check if ``-object`` option behaves as expected"""
+
+def run(self, cmd, *args, **kwargs):
+cmdstr = ' '.join(shlex.quote(c) for c in cmd)
+self.log.info("Running command: %s", cmdstr)
+return subprocess.run(cmd, universal_newlines=True, *args, **kwargs)
+
+def get_devices(self):
+out = self.run([self.qemu_bin, '-object', 'help'],
+   check=True, stdout=subprocess.PIPE).stdout
+lines = out.split('\n')
+return [l.strip() for l in lines[1:] if l.strip()]
+
+@avocado.fail_on(subprocess.CalledProcessError)
+def test_help(self):
+"""Check if ``-object ...,help`` behaves as expected"""
+for device in self.get_devices():
+self.run([self.qemu_bin, '-object', '%s,help' % (device)],
+ check=True,
+ stdout=subprocess.DEVNULL)
+
+@avocado.fail_on(subprocess.CalledProcessError)
+def test_crash(self):
+"""Check that QEMU doesn't crash when using ``-object ...``"""
+for device in self.get_devices():
+r = self.run([self.qemu_bin, '-object',
+'%s,id=obj0' % (device),
+'-monitor', 'stdio'],
+ input='quit\n',
+ stdout=subprocess.DEVNULL,
+ stderr=subprocess.PIPE)
+if r.returncode not in (0, 1):
+self.log.warn("QEMU stderr: %s", r.stderr)
+self.log.warn("QEMU exit code: %d", r.returncode)
+if r.returncode < 0:
+self.fail("QEMU crashed")
+else:
+self.fail("Unexpected exit code")

[PULL 01/39] meson.build: Add comments to clarify code organization

[Paolo Bonzini]

From: Philippe Mathieu-Daudé 

Signed-off-by: Philippe Mathieu-Daudé 
Message-Id: <20201006125602.2311423-2-phi...@redhat.com>
Signed-off-by: Paolo Bonzini 
---
 meson.build | 8 
 1 file changed, 8 insertions(+)

diff --git a/meson.build b/meson.build
index 17c89c87c6..68eaf8ce2f 100644
--- a/meson.build
+++ b/meson.build
@@ -1446,6 +1446,10 @@ qemu_syms = custom_target('qemu.syms', output: 
'qemu.syms',
  capture: true,
  command: [undefsym, nm, '@INPUT@'])
 
+
+# Library dependencies #
+
+
 block_ss = block_ss.apply(config_host, strict: false)
 libblock = static_library('block', block_ss.sources() + genh,
   dependencies: block_ss.dependencies(),
@@ -1465,6 +1469,10 @@ libqmp = static_library('qmp', qmp_ss.sources() + genh,
 
 qmp = declare_dependency(link_whole: [libqmp])
 
+###
+# Targets #
+###
+
 foreach m : block_mods + softmmu_mods
   shared_module(m.name(),
 name_prefix: '',
-- 
2.26.2

[PULL 04/39] chardev: Move the creation of the library to the main meson.build

[Paolo Bonzini]

From: Philippe Mathieu-Daudé 

Be consistent creating all the libraries in the main meson.build file.

Suggested-by: Paolo Bonzini 
Signed-off-by: Philippe Mathieu-Daudé 
Message-Id: <20201006125602.2311423-5-phi...@redhat.com>
Signed-off-by: Paolo Bonzini 
---
 chardev/meson.build | 6 --
 meson.build | 7 +++
 2 files changed, 7 insertions(+), 6 deletions(-)

diff --git a/chardev/meson.build b/chardev/meson.build
index 54e88d0310..dd2699a11b 100644
--- a/chardev/meson.build
+++ b/chardev/meson.build
@@ -1,4 +1,3 @@
-chardev_ss = ss.source_set()
 chardev_ss.add(files(
   'char-fe.c',
   'char-file.c',
@@ -25,11 +24,6 @@ chardev_ss.add(when: 'CONFIG_WIN32', if_true: files(
 ))
 
 chardev_ss = chardev_ss.apply(config_host, strict: false)
-libchardev = static_library('chardev', chardev_ss.sources() + genh,
-name_suffix: 'fa',
-build_by_default: false)
-
-chardev = declare_dependency(link_whole: libchardev)
 
 softmmu_ss.add(files('chardev-sysemu.c', 'msmouse.c', 'wctablet.c', 
'testdev.c'))
 softmmu_ss.add(when: ['CONFIG_SPICE', spice], if_true: files('spice.c'))
diff --git a/meson.build b/meson.build
index 6c760b4163..39fc074977 100644
--- a/meson.build
+++ b/meson.build
@@ -1185,6 +1185,7 @@ sphinx_extn_depends = [ meson.source_root() / 
'docs/sphinx/depfile.py',
 blockdev_ss = ss.source_set()
 block_ss = ss.source_set()
 bsd_user_ss = ss.source_set()
+chardev_ss = ss.source_set()
 common_ss = ss.source_set()
 linux_user_ss = ss.source_set()
 qmp_ss = ss.source_set()
@@ -1469,6 +1470,12 @@ libqmp = static_library('qmp', qmp_ss.sources() + genh,
 
 qmp = declare_dependency(link_whole: [libqmp])
 
+libchardev = static_library('chardev', chardev_ss.sources() + genh,
+name_suffix: 'fa',
+build_by_default: false)
+
+chardev = declare_dependency(link_whole: libchardev)
+
 libhwcore = static_library('hwcore', sources: hwcore_files + genh,
name_suffix: 'fa',
build_by_default: false)
-- 
2.26.2

[PULL 00/39] SCSI, qdev, qtest, meson patches for 2020-10-10

[Paolo Bonzini]

The following changes since commit 4a7c0bd9dcb08798c6f82e55b5a3423f7ee669f1:

  Merge remote-tracking branch 'remotes/dgibson/tags/ppc-for-5.2-20201009' into 
staging (2020-10-09 15:48:04 +0100)

are available in the Git repository at:

  https://gitlab.com/bonzini/qemu.git tags/for-upstream

for you to fetch changes up to 1340ff2adb2624e61c5fcb0eb1889b932b76f669:

  meson: identify more sections of meson.build (2020-10-09 13:19:50 -0400)


* qtest documentation improvements (Eduardo, myself)
* libqtest event buffering (Maxim)
* use RCU for list of children of a bus (Maxim)
* move more files to softmmu/ (myself)
* meson.build cleanups, qemu-storage-daemon fix (Philippe)


Eduardo Habkost (3):
  docs: Move QTest documentation to its own document
  docs/devel/qtest: Include protocol spec in document
  docs/devel/qtest: Include libqtest API reference

Huacai Chen (1):
  meson.build: Re-enable KVM support for MIPS

Marc-André Lureau (1):
  build-sys: fix git version from -version

Maxim Levitsky (11):
  qtest: rename qtest_qmp_receive to qtest_qmp_receive_dict
  qtest: Reintroduce qtest_qmp_receive
  qtest: remove qtest_qmp_receive_success
  qtest: switch users back to qtest_qmp_receive
  scsi/scsi_bus: switch search direction in scsi_device_find
  device_core: use drain_call_rcu in in qmp_device_add
  device-core: use RCU for list of children of a bus
  device-core: use atomic_set on .realized property
  scsi/scsi_bus: Add scsi_device_get
  virtio-scsi: use scsi_device_get
  scsi/scsi_bus: fix races in REPORT LUNS

Paolo Bonzini (13):
  softmmu: move more files to softmmu/
  exec: split out non-softmmu-specific parts
  qom: fix objects with improper parent type
  configure: fix performance regression due to PIC objects
  qtest: unify extra_qtest_srcs and extra_qtest_deps
  docs/devel: update instruction on how to add new unit tests
  device-plug-test: use qtest_qmp to send the device_del command
  qtest: check that drives are really appearing and disappearing
  qemu-iotests, qtest: rewrite test 067 as a qtest
  qdev: add "check if address free" callback for buses
  scsi: switch to bus->check_address
  scsi/scsi-bus: scsi_device_find: don't return unrealized devices
  meson: identify more sections of meson.build

Philippe Mathieu-Daudé (10):
  meson.build: Add comments to clarify code organization
  meson.build: Sort sourcesets alphabetically
  hw/core: Move the creation of the library to the main meson.build
  chardev: Move the creation of the library to the main meson.build
  migration: Move the creation of the library to the main meson.build
  io: Move the creation of the library to the main meson.build
  crypto: Move the creation of the library to the main meson.build
  authz: Move the creation of the library to the main meson.build
  qom: Move the creation of the library to the main meson.build
  hw/nvram: Always register FW_CFG_DATA_GENERATOR_INTERFACE

 .gitlab-ci.yml   |   2 +-
 MAINTAINERS  |  16 +-
 authz/meson.build|  10 -
 chardev/meson.build  |   6 -
 configure|   1 +
 cpu.c| 452 ++
 crypto/meson.build   |  10 -
 docs/devel/index.rst |   1 +
 docs/devel/qtest.rst |  84 ++
 docs/devel/testing.rst   |  64 +
 hw/core/bus.c|  28 +-
 hw/core/meson.build  |   6 -
 hw/core/qdev.c   |  73 +++--
 hw/net/virtio-net.c  |   2 +-
 hw/nvram/fw_cfg-interface.c  |  23 ++
 hw/nvram/fw_cfg.c|   7 -
 hw/nvram/meson.build |   3 +
 hw/scsi/scsi-bus.c   | 262 +++---
 hw/scsi/virtio-scsi.c|  27 +-
 hw/sd/core.c |   3 +-
 include/exec/cpu-common.h|   3 +
 include/hw/acpi/vmgenid.h|   2 +-
 include/hw/misc/vmcoreinfo.h |   2 +-
 include/hw/qdev-core.h   |  24 +-
 include/hw/scsi/scsi.h   |   1 +
 include/net/can_host.h   |   2 +-
 io/meson.build   |  10 -
 meson.build  | 126 +++--
 migration/meson.build|   8 +-
 qom/meson.build  |   8 -
 scripts/coccinelle/qom-parent-type.cocci |  26 ++
 scripts/qemu-version.sh  |   2 +-
 bootdevice.c => softmmu/bootdevice.c |   0
 device_tree.c => softmmu/device_tree.c   |   0
 dma-helpers.c => softmmu/dma-

[PULL 09/39] qom: Move the creation of the library to the main meson.build

[Paolo Bonzini]

From: Philippe Mathieu-Daudé 

Be consistent creating all the libraries in the main meson.build file.

Suggested-by: Paolo Bonzini 
Signed-off-by: Philippe Mathieu-Daudé 
Message-Id: <20201006125602.2311423-10-phi...@redhat.com>
Signed-off-by: Paolo Bonzini 
---
 meson.build | 8 
 qom/meson.build | 8 
 2 files changed, 8 insertions(+), 8 deletions(-)

diff --git a/meson.build b/meson.build
index 2736f74b8f..f4ef3b83f3 100644
--- a/meson.build
+++ b/meson.build
@@ -1192,6 +1192,7 @@ crypto_ss = ss.source_set()
 io_ss = ss.source_set()
 linux_user_ss = ss.source_set()
 qmp_ss = ss.source_set()
+qom_ss = ss.source_set()
 softmmu_ss = ss.source_set()
 specific_fuzz_ss = ss.source_set()
 specific_ss = ss.source_set()
@@ -1454,6 +1455,13 @@ qemu_syms = custom_target('qemu.syms', output: 
'qemu.syms',
 # Library dependencies #
 
 
+qom_ss = qom_ss.apply(config_host, strict: false)
+libqom = static_library('qom', qom_ss.sources() + genh,
+dependencies: [qom_ss.dependencies()],
+name_suffix: 'fa')
+
+qom = declare_dependency(link_whole: libqom)
+
 authz_ss = authz_ss.apply(config_host, strict: false)
 libauthz = static_library('authz', authz_ss.sources() + genh,
   dependencies: [authz_ss.dependencies()],
diff --git a/qom/meson.build b/qom/meson.build
index a1cd03c82c..062a3789d8 100644
--- a/qom/meson.build
+++ b/qom/meson.build
@@ -1,4 +1,3 @@
-qom_ss = ss.source_set()
 qom_ss.add(genh)
 qom_ss.add(files(
   'container.c',
@@ -9,10 +8,3 @@ qom_ss.add(files(
 
 qmp_ss.add(files('qom-qmp-cmds.c'))
 softmmu_ss.add(files('qom-hmp-cmds.c'))
-
-qom_ss = qom_ss.apply(config_host, strict: false)
-libqom = static_library('qom', qom_ss.sources() + genh,
-dependencies: [qom_ss.dependencies()],
-name_suffix: 'fa')
-
-qom = declare_dependency(link_whole: libqom)
-- 
2.26.2

[PULL 05/39] migration: Move the creation of the library to the main meson.build

[Paolo Bonzini]

From: Philippe Mathieu-Daudé 

Be consistent creating all the libraries in the main meson.build file.

Suggested-by: Paolo Bonzini 
Signed-off-by: Philippe Mathieu-Daudé 
Message-Id: <20201006125602.2311423-6-phi...@redhat.com>
Signed-off-by: Paolo Bonzini 
---
 meson.build   | 7 +++
 migration/meson.build | 8 +---
 2 files changed, 8 insertions(+), 7 deletions(-)

diff --git a/meson.build b/meson.build
index 39fc074977..dce48d8304 100644
--- a/meson.build
+++ b/meson.build
@@ -1451,6 +1451,13 @@ qemu_syms = custom_target('qemu.syms', output: 
'qemu.syms',
 # Library dependencies #
 
 
+libmigration = static_library('migration', sources: migration_files + genh,
+  name_suffix: 'fa',
+  build_by_default: false)
+migration = declare_dependency(link_with: libmigration,
+   dependencies: [zlib, qom, io])
+softmmu_ss.add(migration)
+
 block_ss = block_ss.apply(config_host, strict: false)
 libblock = static_library('block', block_ss.sources() + genh,
   dependencies: block_ss.dependencies(),
diff --git a/migration/meson.build b/migration/meson.build
index b5b71c8060..980e37865c 100644
--- a/migration/meson.build
+++ b/migration/meson.build
@@ -8,13 +8,7 @@ migration_files = files(
   'qemu-file.c',
   'qjson.c',
 )
-
-libmigration = static_library('migration', sources: migration_files + genh,
-  name_suffix: 'fa',
-  build_by_default: false)
-migration = declare_dependency(link_with: libmigration,
-   dependencies: [zlib, qom, io])
-softmmu_ss.add(migration)
+softmmu_ss.add(migration_files)
 
 softmmu_ss.add(files(
   'block-dirty-bitmap.c',
-- 
2.26.2

[PULL 02/39] meson.build: Sort sourcesets alphabetically

[Paolo Bonzini]

From: Philippe Mathieu-Daudé 

Signed-off-by: Philippe Mathieu-Daudé 
Message-Id: <20201006125602.2311423-3-phi...@redhat.com>
Signed-off-by: Paolo Bonzini 
---
 meson.build | 18 +-
 1 file changed, 9 insertions(+), 9 deletions(-)

diff --git a/meson.build b/meson.build
index 68eaf8ce2f..adcfb494a8 100644
--- a/meson.build
+++ b/meson.build
@@ -1182,19 +1182,19 @@ sphinx_extn_depends = [ meson.source_root() / 
'docs/sphinx/depfile.py',
 
 # Collect sourcesets.
 
-util_ss = ss.source_set()
-stub_ss = ss.source_set()
-trace_ss = ss.source_set()
-block_ss = ss.source_set()
 blockdev_ss = ss.source_set()
-qmp_ss = ss.source_set()
-common_ss = ss.source_set()
-softmmu_ss = ss.source_set()
-user_ss = ss.source_set()
+block_ss = ss.source_set()
 bsd_user_ss = ss.source_set()
+common_ss = ss.source_set()
 linux_user_ss = ss.source_set()
-specific_ss = ss.source_set()
+qmp_ss = ss.source_set()
+softmmu_ss = ss.source_set()
 specific_fuzz_ss = ss.source_set()
+specific_ss = ss.source_set()
+stub_ss = ss.source_set()
+trace_ss = ss.source_set()
+user_ss = ss.source_set()
+util_ss = ss.source_set()
 
 modules = {}
 hw_arch = {}
-- 
2.26.2

[PULL 16/39] docs/devel/qtest: Include protocol spec in document

[Paolo Bonzini]

From: Eduardo Habkost 

Include the QTest Protocol doc string in docs/devel/qtest.rst,
after converting it to use Sphinx syntax.

Signed-off-by: Eduardo Habkost 
Acked-by: Thomas Huth 
Message-Id: <20201005205228.697463-3-ehabk...@redhat.com>
Signed-off-by: Paolo Bonzini 
---
 docs/devel/qtest.rst | 12 ++--
 softmmu/qtest.c  | 71 +++-
 2 files changed, 73 insertions(+), 10 deletions(-)

diff --git a/docs/devel/qtest.rst b/docs/devel/qtest.rst
index 86dec84a0b..3bf9ebee7f 100644
--- a/docs/devel/qtest.rst
+++ b/docs/devel/qtest.rst
@@ -4,8 +4,8 @@ QTest Device Emulation Testing Framework
 
 QTest is a device emulation testing framework.  It can be very useful to test
 device models; it could also control certain aspects of QEMU (such as virtual
-clock stepping), with a special purpose "qtest" protocol.  Refer to the
-documentation in ``qtest.c`` for more details of the protocol.
+clock stepping), with a special purpose "qtest" protocol.  Refer to
+:ref:`qtest-protocol` for more details of the protocol.
 
 QTest cases can be executed with
 
@@ -56,3 +56,11 @@ from the output of
 
 which you can run manually.
 
+
+.. _qtest-protocol:
+
+QTest Protocol
+--
+
+.. kernel-doc:: softmmu/qtest.c
+   :doc: QTest Protocol
diff --git a/softmmu/qtest.c b/softmmu/qtest.c
index 0d43cf8883..2c6e8dc858 100644
--- a/softmmu/qtest.c
+++ b/softmmu/qtest.c
@@ -49,92 +49,139 @@ static void *qtest_server_send_opaque;
 #define FMT_timeval "%ld.%06ld"
 
 /**
- * QTest Protocol
+ * DOC: QTest Protocol
  *
  * Line based protocol, request/response based.  Server can send async messages
  * so clients should always handle many async messages before the response
  * comes in.
  *
  * Valid requests
+ * ^^
  *
  * Clock management:
+ * "
  *
  * The qtest client is completely in charge of the QEMU_CLOCK_VIRTUAL.  qtest 
commands
  * let you adjust the value of the clock (monotonically).  All the commands
  * return the current value of the clock in nanoseconds.
  *
+ * .. code-block:: none
+ *
  *  > clock_step
  *  < OK VALUE
  *
- * Advance the clock to the next deadline.  Useful when waiting for
- * asynchronous events.
+ * Advance the clock to the next deadline.  Useful when waiting for
+ * asynchronous events.
+ *
+ * .. code-block:: none
  *
  *  > clock_step NS
  *  < OK VALUE
  *
- * Advance the clock by NS nanoseconds.
+ * Advance the clock by NS nanoseconds.
+ *
+ * .. code-block:: none
  *
  *  > clock_set NS
  *  < OK VALUE
  *
- * Advance the clock to NS nanoseconds (do nothing if it's already past).
+ * Advance the clock to NS nanoseconds (do nothing if it's already past).
  *
  * PIO and memory access:
+ * ""
+ *
+ * .. code-block:: none
  *
  *  > outb ADDR VALUE
  *  < OK
  *
+ * .. code-block:: none
+ *
  *  > outw ADDR VALUE
  *  < OK
  *
+ * .. code-block:: none
+ *
  *  > outl ADDR VALUE
  *  < OK
  *
+ * .. code-block:: none
+ *
  *  > inb ADDR
  *  < OK VALUE
  *
+ * .. code-block:: none
+ *
  *  > inw ADDR
  *  < OK VALUE
  *
+ * .. code-block:: none
+ *
  *  > inl ADDR
  *  < OK VALUE
  *
+ * .. code-block:: none
+ *
  *  > writeb ADDR VALUE
  *  < OK
  *
+ * .. code-block:: none
+ *
  *  > writew ADDR VALUE
  *  < OK
  *
+ * .. code-block:: none
+ *
  *  > writel ADDR VALUE
  *  < OK
  *
+ * .. code-block:: none
+ *
  *  > writeq ADDR VALUE
  *  < OK
  *
+ * .. code-block:: none
+ *
  *  > readb ADDR
  *  < OK VALUE
  *
+ * .. code-block:: none
+ *
  *  > readw ADDR
  *  < OK VALUE
  *
+ * .. code-block:: none
+ *
  *  > readl ADDR
  *  < OK VALUE
  *
+ * .. code-block:: none
+ *
  *  > readq ADDR
  *  < OK VALUE
  *
+ * .. code-block:: none
+ *
  *  > read ADDR SIZE
  *  < OK DATA
  *
+ * .. code-block:: none
+ *
  *  > write ADDR SIZE DATA
  *  < OK
  *
+ * .. code-block:: none
+ *
  *  > b64read ADDR SIZE
  *  < OK B64_DATA
  *
+ * .. code-block:: none
+ *
  *  > b64write ADDR SIZE B64_DATA
  *  < OK
  *
+ * .. code-block:: none
+ *
  *  > memset ADDR SIZE VALUE
  *  < OK
  *
@@ -149,16 +196,21 @@ static void *qtest_server_send_opaque;
  * If the sizes do not match, the data will be truncated.
  *
  * IRQ management:
+ * """
+ *
+ * .. code-block:: none
  *
  *  > irq_intercept_in QOM-PATH
  *  < OK
  *
+ * .. code-block:: none
+ *
  *  > irq_intercept_out QOM-PATH
  *  < OK
  *
  * Attach to the gpio-in (resp. gpio-out) pins exported by the device at
  * QOM-PATH.  When the pin is triggered, one of the following async messages
- * will be printed to the qtest stream:
+ * will be printed to the qtest stream::
  *
  *  IRQ raise NUM
  *  IRQ lower NUM
@@ -168,12 +220,15 @@ static void *qtest_server_send_opaque;
  * NUM=0 even though it is remapped to GSI 2).
  *
  * Setting interrupt level:
+ * 
+ *
+ * .. code-block:: none
  *
  *  > set_irq_in QOM-PATH NAME NUM LEVEL
  *  < OK
  *
- *  where NAME is the name of the irq/gpio list, NUM is an IRQ numbe

[PULL 19/39] docs/devel: update instruction on how to add new unit tests

[Paolo Bonzini]

Signed-off-by: Paolo Bonzini 
---
 docs/devel/qtest.rst   | 30 +-
 docs/devel/testing.rst | 19 ++-
 2 files changed, 31 insertions(+), 18 deletions(-)

diff --git a/docs/devel/qtest.rst b/docs/devel/qtest.rst
index 075fe5f7d5..97c5a75626 100644
--- a/docs/devel/qtest.rst
+++ b/docs/devel/qtest.rst
@@ -33,15 +33,27 @@ Steps to add a new QTest case are:
 2. Write the test code with the glib and libqtest/libqos API. See also existing
tests and the library headers for reference.
 
-3. Register the new test in ``tests/qtest/Makefile.include``. Add the test
-   executable name to an appropriate ``check-qtest-*-y`` variable. For example:
-
-   ``check-qtest-generic-y = tests/qtest/foo-test$(EXESUF)``
-
-4. Add object dependencies of the executable in the Makefile, including the
-   test source file(s) and other interesting objects. For example:
-
-   ``tests/qtest/foo-test$(EXESUF): tests/qtest/foo-test.o $(libqos-obj-y)``
+3. Register the new test in ``tests/qtest/meson.build``. Add the test
+   executable name to an appropriate ``qtests_*`` variable. There is
+   one variable per architecture, plus ``qtests_generic`` for tests
+   that can be run for all architectures.  For example::
+
+ qtests_generic = [
+   ...
+   'foo-test',
+   ...
+ ]
+
+4. If the test has more than one source file or needs to be linked with any
+   dependency other than ``qemuutil`` and ``qos``, list them in the ``qtests``
+   dictionary.  For example a test that needs to use the ``QIO`` library
+   will have an entry like::
+
+ {
+   ...
+   'foo-test': [io],
+   ...
+ }
 
 Debugging a QTest failure is slightly harder than the unit test because the
 tests look up QEMU program names in the environment variables, such as
diff --git a/docs/devel/testing.rst b/docs/devel/testing.rst
index a171494b4e..cecee6eaa1 100644
--- a/docs/devel/testing.rst
+++ b/docs/devel/testing.rst
@@ -41,15 +41,16 @@ add a new unit test:
test. The test code should be organized with the glib testing framework.
Copying and modifying an existing test is usually a good idea.
 
-3. Add the test to ``tests/Makefile.include``. First, name the unit test
-   program and add it to ``$(check-unit-y)``; then add a rule to build the
-   executable.  For example:
-
-.. code::
-
-  check-unit-y += tests/foo-test$(EXESUF)
-  tests/foo-test$(EXESUF): tests/foo-test.o $(test-util-obj-y)
-  ...
+3. Add the test to ``tests/meson.build``. The unit tests are listed in a
+   dictionary called ``tests``.  The values are any additional sources and
+   dependencies to be linked with the test.  For a simple test whose source
+   is in ``tests/foo-test.c``, it is enough to add an entry like::
+
+ {
+   ...
+   'foo-test': [],
+   ...
+ }
 
 Since unit tests don't require environment variables, the simplest way to debug
 a unit test failure is often directly invoking it or even running it under
-- 
2.26.2

[PULL 18/39] qtest: unify extra_qtest_srcs and extra_qtest_deps

[Paolo Bonzini]

Currently the extra sources and extra dependencies of qtests are held
in two separate dictionaries.  Use the same trick as tests/meson.build
to combine them into one.  This will make it easier to update the
documentation for unit tests and qtests.

Signed-off-by: Paolo Bonzini 
---
 tests/qtest/meson.build | 55 +++--
 1 file changed, 26 insertions(+), 29 deletions(-)

diff --git a/tests/qtest/meson.build b/tests/qtest/meson.build
index 0f32ca0895..28bafc02a2 100644
--- a/tests/qtest/meson.build
+++ b/tests/qtest/meson.build
@@ -193,35 +193,25 @@ qos_test_ss.add(
 qos_test_ss.add(when: 'CONFIG_VIRTFS', if_true: files('virtio-9p-test.c'))
 qos_test_ss.add(when: 'CONFIG_VHOST_USER', if_true: files('vhost-user-test.c'))
 
-extra_qtest_deps = {
-  'bios-tables-test': [io],
-  'ivshmem-test': [rt],
-  'qos-test': [chardev, io],
-  'tpm-crb-swtpm-test': [io],
-  'tpm-crb-test': [io],
-  'tpm-tis-swtpm-test': [io],
-  'tpm-tis-test': [io],
-  'tpm-tis-device-swtpm-test': [io],
-  'tpm-tis-device-test': [io],
-}
-extra_qtest_srcs = {
-  'bios-tables-test': files('boot-sector.c', 'acpi-utils.c', 'tpm-emu.c'),
-  'pxe-test': files('boot-sector.c'),
+tpmemu_files = ['tpm-emu.c', 'tpm-util.c', 'tpm-tests.c']
+
+qtests = {
+  'bios-tables-test': [io, 'boot-sector.c', 'acpi-utils.c', 'tpm-emu.c'],
   'cdrom-test': files('boot-sector.c'),
-  'migration-test': files('migration-helpers.c'),
-  'ivshmem-test': files('../../contrib/ivshmem-server/ivshmem-server.c'),
   'dbus-vmstate-test': files('migration-helpers.c') + dbus_vmstate1,
+  'ivshmem-test': [rt, '../../contrib/ivshmem-server/ivshmem-server.c'],
+  'migration-test': files('migration-helpers.c'),
+  'pxe-test': files('boot-sector.c'),
+  'qos-test': [chardev, io, qos_test_ss.apply(config_host, strict: 
false).sources()],
+  'tpm-crb-swtpm-test': [io, tpmemu_files],
+  'tpm-crb-test': [io, tpmemu_files],
+  'tpm-tis-swtpm-test': [io, tpmemu_files, 'tpm-tis-util.c'],
+  'tpm-tis-test': [io, tpmemu_files, 'tpm-tis-util.c'],
+  'tpm-tis-device-swtpm-test': [io, tpmemu_files, 'tpm-tis-util.c'],
+  'tpm-tis-device-test': [io, tpmemu_files, 'tpm-tis-util.c'],
   'vmgenid-test': files('boot-sector.c', 'acpi-utils.c'),
-  'tpm-crb-swtpm-test': files('tpm-emu.c', 'tpm-util.c', 'tpm-tests.c'),
-  'tpm-crb-test': files('tpm-emu.c', 'tpm-util.c', 'tpm-tests.c'),
-  'tpm-tis-device-swtpm-test': files('tpm-emu.c', 'tpm-util.c', 
'tpm-tis-util.c', 'tpm-tests.c'),
-  'tpm-tis-device-test': files('tpm-emu.c', 'tpm-util.c', 'tpm-tis-util.c', 
'tpm-tests.c'),
-  'tpm-tis-swtpm-test': files('tpm-emu.c', 'tpm-util.c', 'tpm-tis-util.c', 
'tpm-tests.c'),
-  'tpm-tis-test': files('tpm-emu.c', 'tpm-util.c', 'tpm-tis-util.c', 
'tpm-tests.c'),
-  'qos-test': qos_test_ss.apply(config_host, strict: false).sources()
 }
 
-
 qtest_executables = {}
 foreach dir : target_dirs
   if not dir.endswith('-softmmu')
@@ -230,7 +220,7 @@ foreach dir : target_dirs
 
   target_base = dir.split('-')[0]
   qtest_emulator = emulators['qemu-system-' + target_base]
-  qtests = get_variable('qtests_' + target_base, []) + qtests_generic
+  target_qtests = get_variable('qtests_' + target_base, []) + qtests_generic
 
   test_deps = []
   qtest_env = environment()
@@ -241,14 +231,21 @@ foreach dir : target_dirs
   qtest_env.set('G_TEST_DBUS_DAEMON', meson.source_root() / 
'tests/dbus-vmstate-daemon.sh')
   qtest_env.set('QTEST_QEMU_BINARY', './qemu-system-' + target_base)
   
-  foreach test : qtests
+  foreach test : target_qtests
 # Executables are shared across targets, declare them only the first time 
we
 # encounter them
 if not qtest_executables.has_key(test)
+  src = [test + '.c']
+  deps = [qemuutil, qos]
+  if test in qtests
+# use a sourceset to quickly separate sources and deps
+test_ss = ss.source_set()
+test_ss.add(qtests[test])
+src += test_ss.all_sources()
+deps += test_ss.all_dependencies()
+  endif
   qtest_executables += {
-test: executable(test,
- files(test + '.c') + extra_qtest_srcs.get(test, []),
- dependencies: [qemuutil, qos] + 
extra_qtest_deps.get(test, []))
+test: executable(test, src, dependencies: deps)
   }
 endif
 # FIXME: missing dependency on the emulator binary and qemu-img
-- 
2.26.2

[PULL 03/39] hw/core: Move the creation of the library to the main meson.build

[Paolo Bonzini]

From: Philippe Mathieu-Daudé 

Be consistent creating all the libraries in the main meson.build file.

Suggested-by: Paolo Bonzini 
Signed-off-by: Philippe Mathieu-Daudé 
Message-Id: <20201006125602.2311423-4-phi...@redhat.com>
Signed-off-by: Paolo Bonzini 
---
 hw/core/meson.build | 6 --
 meson.build | 6 ++
 2 files changed, 6 insertions(+), 6 deletions(-)

diff --git a/hw/core/meson.build b/hw/core/meson.build
index fc91f98075..4a744f3b5e 100644
--- a/hw/core/meson.build
+++ b/hw/core/meson.build
@@ -14,12 +14,6 @@ hwcore_files = files(
   'qdev-clock.c',
 )
 
-libhwcore = static_library('hwcore', sources: hwcore_files + genh,
-   name_suffix: 'fa',
-   build_by_default: false)
-hwcore = declare_dependency(link_whole: libhwcore)
-common_ss.add(hwcore)
-
 common_ss.add(files('cpu.c'))
 common_ss.add(when: 'CONFIG_FITLOADER', if_true: files('loader-fit.c'))
 common_ss.add(when: 'CONFIG_GENERIC_LOADER', if_true: 
files('generic-loader.c'))
diff --git a/meson.build b/meson.build
index adcfb494a8..6c760b4163 100644
--- a/meson.build
+++ b/meson.build
@@ -1469,6 +1469,12 @@ libqmp = static_library('qmp', qmp_ss.sources() + genh,
 
 qmp = declare_dependency(link_whole: [libqmp])
 
+libhwcore = static_library('hwcore', sources: hwcore_files + genh,
+   name_suffix: 'fa',
+   build_by_default: false)
+hwcore = declare_dependency(link_whole: libhwcore)
+common_ss.add(hwcore)
+
 ###
 # Targets #
 ###
-- 
2.26.2

[PULL 15/39] docs: Move QTest documentation to its own document

[Paolo Bonzini]

From: Eduardo Habkost 

The qtest and libqtest doc comments will be parsed to generate
API documentation, so move QTest documentation to its own
document where the API and format documentation and will be
included.

Signed-off-by: Eduardo Habkost 
Acked-by: Thomas Huth 
Message-Id: <20201005205228.697463-2-ehabk...@redhat.com>
Signed-off-by: Paolo Bonzini 
---
 docs/devel/index.rst   |  1 +
 docs/devel/qtest.rst   | 58 ++
 docs/devel/testing.rst | 47 ++
 3 files changed, 61 insertions(+), 45 deletions(-)
 create mode 100644 docs/devel/qtest.rst

diff --git a/docs/devel/index.rst b/docs/devel/index.rst
index 5fda2d3509..77baae5c77 100644
--- a/docs/devel/index.rst
+++ b/docs/devel/index.rst
@@ -21,6 +21,7 @@ Contents:
atomics
stable-process
testing
+   qtest
decodetree
secure-coding-practices
tcg
diff --git a/docs/devel/qtest.rst b/docs/devel/qtest.rst
new file mode 100644
index 00..86dec84a0b
--- /dev/null
+++ b/docs/devel/qtest.rst
@@ -0,0 +1,58 @@
+
+QTest Device Emulation Testing Framework
+
+
+QTest is a device emulation testing framework.  It can be very useful to test
+device models; it could also control certain aspects of QEMU (such as virtual
+clock stepping), with a special purpose "qtest" protocol.  Refer to the
+documentation in ``qtest.c`` for more details of the protocol.
+
+QTest cases can be executed with
+
+.. code::
+
+   make check-qtest
+
+The QTest library is implemented by ``tests/qtest/libqtest.c`` and the API is
+defined in ``tests/qtest/libqtest.h``.
+
+Consider adding a new QTest case when you are introducing a new virtual
+hardware, or extending one if you are adding functionalities to an existing
+virtual device.
+
+On top of libqtest, a higher level library, ``libqos``, was created to
+encapsulate common tasks of device drivers, such as memory management and
+communicating with system buses or devices. Many virtual device tests use
+libqos instead of directly calling into libqtest.
+
+Steps to add a new QTest case are:
+
+1. Create a new source file for the test. (More than one file can be added as
+   necessary.) For example, ``tests/qtest/foo-test.c``.
+
+2. Write the test code with the glib and libqtest/libqos API. See also existing
+   tests and the library headers for reference.
+
+3. Register the new test in ``tests/qtest/Makefile.include``. Add the test
+   executable name to an appropriate ``check-qtest-*-y`` variable. For example:
+
+   ``check-qtest-generic-y = tests/qtest/foo-test$(EXESUF)``
+
+4. Add object dependencies of the executable in the Makefile, including the
+   test source file(s) and other interesting objects. For example:
+
+   ``tests/qtest/foo-test$(EXESUF): tests/qtest/foo-test.o $(libqos-obj-y)``
+
+Debugging a QTest failure is slightly harder than the unit test because the
+tests look up QEMU program names in the environment variables, such as
+``QTEST_QEMU_BINARY`` and ``QTEST_QEMU_IMG``, and also because it is not easy
+to attach gdb to the QEMU process spawned from the test. But manual invoking
+and using gdb on the test is still simple to do: find out the actual command
+from the output of
+
+.. code::
+
+  make check-qtest V=1
+
+which you can run manually.
+
diff --git a/docs/devel/testing.rst b/docs/devel/testing.rst
index bd64c1bdcd..a171494b4e 100644
--- a/docs/devel/testing.rst
+++ b/docs/devel/testing.rst
@@ -70,8 +70,8 @@ QTest
 
 QTest is a device emulation testing framework.  It can be very useful to test
 device models; it could also control certain aspects of QEMU (such as virtual
-clock stepping), with a special purpose "qtest" protocol.  Refer to the
-documentation in ``qtest.c`` for more details of the protocol.
+clock stepping), with a special purpose "qtest" protocol.  Refer to
+:doc:`qtest` for more details.
 
 QTest cases can be executed with
 
@@ -79,49 +79,6 @@ QTest cases can be executed with
 
make check-qtest
 
-The QTest library is implemented by ``tests/qtest/libqtest.c`` and the API is
-defined in ``tests/qtest/libqtest.h``.
-
-Consider adding a new QTest case when you are introducing a new virtual
-hardware, or extending one if you are adding functionalities to an existing
-virtual device.
-
-On top of libqtest, a higher level library, ``libqos``, was created to
-encapsulate common tasks of device drivers, such as memory management and
-communicating with system buses or devices. Many virtual device tests use
-libqos instead of directly calling into libqtest.
-
-Steps to add a new QTest case are:
-
-1. Create a new source file for the test. (More than one file can be added as
-   necessary.) For example, ``tests/qtest/foo-test.c``.
-
-2. Write the test code with the glib and libqtest/libqos API. See also existing
-   tests and the library headers for reference.
-
-3. Register the new test in ``tests/qtest/Makefile.include``. Add the

[PULL 23/39] qtest: Reintroduce qtest_qmp_receive

[Paolo Bonzini]

From: Maxim Levitsky 

The new qtest_qmp_receive buffers all the received qmp events, allowing
qtest_qmp_eventwait_ref to return them.

This is intended to solve the race in regard to ordering of qmp events
vs qmp responses, as soon as the callers start using the new interface.

In addition to that, define qtest_qmp_event_ref a function which only scans
the buffer that qtest_qmp_receive stores the events to.

This is intended for callers that are only interested in events that were
received during the last call to the qtest_qmp_receive.

Suggested-by: Paolo Bonzini 
Signed-off-by: Maxim Levitsky 
Message-Id: <20201006123904.610658-3-mlevi...@redhat.com>
Signed-off-by: Paolo Bonzini 
---
 tests/qtest/libqos/libqtest.h | 23 
 tests/qtest/libqtest.c| 49 ++-
 2 files changed, 71 insertions(+), 1 deletion(-)

diff --git a/tests/qtest/libqos/libqtest.h b/tests/qtest/libqos/libqtest.h
index 9b3f99b322..a2e3961792 100644
--- a/tests/qtest/libqos/libqtest.h
+++ b/tests/qtest/libqos/libqtest.h
@@ -198,6 +198,16 @@ void qtest_qmp_vsend(QTestState *s, const char *fmt, 
va_list ap)
  */
 QDict *qtest_qmp_receive_dict(QTestState *s);
 
+/**
+ * qtest_qmp_receive:
+ * @s: #QTestState instance to operate on.
+ *
+ * Reads a QMP message from QEMU and returns the response.
+ * Buffers all the events received meanwhile, until a
+ * call to qtest_qmp_eventwait
+ */
+QDict *qtest_qmp_receive(QTestState *s);
+
 /**
  * qtest_qmp_eventwait:
  * @s: #QTestState instance to operate on.
@@ -217,6 +227,19 @@ void qtest_qmp_eventwait(QTestState *s, const char *event);
  */
 QDict *qtest_qmp_eventwait_ref(QTestState *s, const char *event);
 
+/**
+ * qtest_qmp_event_ref:
+ * @s: #QTestState instance to operate on.
+ * @s: #event event to return.
+ *
+ * Removes non-matching events from the buffer that was set by
+ * qtest_qmp_receive, until an event bearing the given name is found,
+ * and returns it.
+ * If no event matches, clears the buffer and returns NULL.
+ *
+ */
+QDict *qtest_qmp_event_ref(QTestState *s, const char *event);
+
 /**
  * qtest_qmp_receive_success:
  * @s: #QTestState instance to operate on
diff --git a/tests/qtest/libqtest.c b/tests/qtest/libqtest.c
index dadc465825..d4c49a52ff 100644
--- a/tests/qtest/libqtest.c
+++ b/tests/qtest/libqtest.c
@@ -63,6 +63,7 @@ struct QTestState
 bool irq_level[MAX_IRQ];
 GString *rx;
 QTestTransportOps ops;
+GList *pending_events;
 };
 
 static GHookList abrt_hooks;
@@ -279,6 +280,7 @@ QTestState *qtest_init_without_qmp_handshake(const char 
*extra_args)
 
 g_test_message("starting QEMU: %s", command);
 
+s->pending_events = NULL;
 s->wstatus = 0;
 s->expected_status = 0;
 s->qemu_pid = fork();
@@ -386,6 +388,13 @@ void qtest_quit(QTestState *s)
 close(s->fd);
 close(s->qmp_fd);
 g_string_free(s->rx, true);
+
+for (GList *it = s->pending_events; it != NULL; it = it->next) {
+qobject_unref((QDict *)it->data);
+}
+
+g_list_free(s->pending_events);
+
 g_free(s);
 }
 
@@ -603,6 +612,19 @@ QDict *qmp_fd_receive(int fd)
 return qmp.response;
 }
 
+QDict *qtest_qmp_receive(QTestState *s)
+{
+while (true) {
+QDict *response = qtest_qmp_receive_dict(s);
+
+if (!qdict_get_try_str(response, "event")) {
+return response;
+}
+/* Stash the event for a later consumption */
+s->pending_events = g_list_prepend(s->pending_events, response);
+}
+}
+
 QDict *qtest_qmp_receive_dict(QTestState *s)
 {
 return qmp_fd_receive(s->qmp_fd);
@@ -771,10 +793,34 @@ void qtest_qmp_send_raw(QTestState *s, const char *fmt, 
...)
 va_end(ap);
 }
 
-QDict *qtest_qmp_eventwait_ref(QTestState *s, const char *event)
+QDict *qtest_qmp_event_ref(QTestState *s, const char *event)
 {
+GList *next = NULL;
 QDict *response;
 
+for (GList *it = s->pending_events; it != NULL; it = next) {
+
+next = it->next;
+response = (QDict *)it->data;
+
+s->pending_events = g_list_remove_link(s->pending_events, it);
+
+if (!strcmp(qdict_get_str(response, "event"), event)) {
+return response;
+}
+qobject_unref(response);
+}
+return NULL;
+}
+
+QDict *qtest_qmp_eventwait_ref(QTestState *s, const char *event)
+{
+QDict *response = qtest_qmp_event_ref(s, event);
+
+if (response) {
+return response;
+}
+
 for (;;) {
 response = qtest_qmp_receive_dict(s);
 if ((qdict_haskey(response, "event")) &&
@@ -1403,6 +1449,7 @@ QTestState *qtest_inproc_init(QTestState **s, bool log, 
const char* arch,
 {
 QTestState *qts;
 qts = g_new0(QTestState, 1);
+qts->pending_events = NULL;
 *s = qts; /* Expose qts early on, since the query endianness relies on it 
*/
 qts->wstatus = 0;
 for (int i = 0; i < MAX_IRQ; i++) {
-- 
2.26.2

[PULL 06/39] io: Move the creation of the library to the main meson.build

[Paolo Bonzini]

From: Philippe Mathieu-Daudé 

Be consistent creating all the libraries in the main meson.build file.

Suggested-by: Paolo Bonzini 
Signed-off-by: Philippe Mathieu-Daudé 
Message-Id: <20201006125602.2311423-7-phi...@redhat.com>
Signed-off-by: Paolo Bonzini 
---
 io/meson.build | 10 --
 meson.build| 10 ++
 2 files changed, 10 insertions(+), 10 deletions(-)

diff --git a/io/meson.build b/io/meson.build
index 768c1b5ec3..bcd8b1e737 100644
--- a/io/meson.build
+++ b/io/meson.build
@@ -1,4 +1,3 @@
-io_ss = ss.source_set()
 io_ss.add(genh)
 io_ss.add(files(
   'channel-buffer.c',
@@ -14,12 +13,3 @@ io_ss.add(files(
   'net-listener.c',
   'task.c',
 ))
-
-io_ss = io_ss.apply(config_host, strict: false)
-libio = static_library('io', io_ss.sources() + genh,
-   dependencies: [io_ss.dependencies()],
-   link_with: libqemuutil,
-   name_suffix: 'fa',
-   build_by_default: false)
-
-io = declare_dependency(link_whole: libio, dependencies: [crypto, qom])
diff --git a/meson.build b/meson.build
index dce48d8304..28f31e6022 100644
--- a/meson.build
+++ b/meson.build
@@ -1187,6 +1187,7 @@ block_ss = ss.source_set()
 bsd_user_ss = ss.source_set()
 chardev_ss = ss.source_set()
 common_ss = ss.source_set()
+io_ss = ss.source_set()
 linux_user_ss = ss.source_set()
 qmp_ss = ss.source_set()
 softmmu_ss = ss.source_set()
@@ -1451,6 +1452,15 @@ qemu_syms = custom_target('qemu.syms', output: 
'qemu.syms',
 # Library dependencies #
 
 
+io_ss = io_ss.apply(config_host, strict: false)
+libio = static_library('io', io_ss.sources() + genh,
+   dependencies: [io_ss.dependencies()],
+   link_with: libqemuutil,
+   name_suffix: 'fa',
+   build_by_default: false)
+
+io = declare_dependency(link_whole: libio, dependencies: [crypto, qom])
+
 libmigration = static_library('migration', sources: migration_files + genh,
   name_suffix: 'fa',
   build_by_default: false)
-- 
2.26.2

[PULL 11/39] softmmu: move more files to softmmu/

[Paolo Bonzini]

Keep most softmmu_ss files into the system-emulation-specific
directory.

Signed-off-by: Paolo Bonzini 
---
 MAINTAINERS  |  9 +
 meson.build  | 10 --
 bootdevice.c => softmmu/bootdevice.c |  0
 device_tree.c => softmmu/device_tree.c   |  0
 dma-helpers.c => softmmu/dma-helpers.c   |  0
 softmmu/meson.build  | 10 ++
 qdev-monitor.c => softmmu/qdev-monitor.c |  0
 qemu-seccomp.c => softmmu/qemu-seccomp.c |  0
 tpm.c => softmmu/tpm.c   |  0
 9 files changed, 15 insertions(+), 14 deletions(-)
 rename bootdevice.c => softmmu/bootdevice.c (100%)
 rename device_tree.c => softmmu/device_tree.c (100%)
 rename dma-helpers.c => softmmu/dma-helpers.c (100%)
 rename qdev-monitor.c => softmmu/qdev-monitor.c (100%)
 rename qemu-seccomp.c => softmmu/qemu-seccomp.c (100%)
 rename tpm.c => softmmu/tpm.c (100%)

diff --git a/MAINTAINERS b/MAINTAINERS
index 8d50b96c33..dda54f000d 100644
--- a/MAINTAINERS
+++ b/MAINTAINERS
@@ -2235,7 +2235,7 @@ Device Tree
 M: Alistair Francis 
 R: David Gibson 
 S: Maintained
-F: device_tree.c
+F: softmmu/device_tree.c
 F: include/sysemu/device_tree.h
 
 Dump
@@ -2281,6 +2281,7 @@ F: include/exec/memop.h
 F: include/exec/memory.h
 F: include/exec/ram_addr.h
 F: include/exec/ramblock.h
+F: softmmu/dma-helpers.c
 F: softmmu/ioport.c
 F: softmmu/memory.c
 F: include/exec/memory-internal.h
@@ -2461,7 +2462,7 @@ F: include/monitor/qdev.h
 F: include/qom/
 F: qapi/qom.json
 F: qapi/qdev.json
-F: qdev-monitor.c
+F: softmmu/qdev-monitor.c
 F: qom/
 F: tests/check-qom-interface.c
 F: tests/check-qom-proplist.c
@@ -2591,7 +2592,7 @@ F: docs/interop/dbus-vmstate.rst
 Seccomp
 M: Eduardo Otubo 
 S: Supported
-F: qemu-seccomp.c
+F: softmmu/qemu-seccomp.c
 F: include/sysemu/seccomp.h
 
 Cryptography
@@ -2957,7 +2958,7 @@ T: git https://github.com/stefanha/qemu.git block
 Bootdevice
 M: Gonglei 
 S: Maintained
-F: bootdevice.c
+F: softmmu/bootdevice.c
 
 Quorum
 M: Alberto Garcia 
diff --git a/meson.build b/meson.build
index f4ef3b83f3..a280e4cf21 100644
--- a/meson.build
+++ b/meson.build
@@ -1365,17 +1365,7 @@ blockdev_ss.add(files(
 # os-win32.c does not
 blockdev_ss.add(when: 'CONFIG_POSIX', if_true: files('os-posix.c'))
 softmmu_ss.add(when: 'CONFIG_WIN32', if_true: [files('os-win32.c')])
-
 softmmu_ss.add_all(blockdev_ss)
-softmmu_ss.add(files(
-  'bootdevice.c',
-  'dma-helpers.c',
-  'qdev-monitor.c',
-), sdl)
-
-softmmu_ss.add(when: 'CONFIG_TPM', if_true: files('tpm.c'))
-softmmu_ss.add(when: 'CONFIG_SECCOMP', if_true: [files('qemu-seccomp.c'), 
seccomp])
-softmmu_ss.add(when: fdt, if_true: files('device_tree.c'))
 
 common_ss.add(files('cpus-common.c'))
 
diff --git a/bootdevice.c b/softmmu/bootdevice.c
similarity index 100%
rename from bootdevice.c
rename to softmmu/bootdevice.c
diff --git a/device_tree.c b/softmmu/device_tree.c
similarity index 100%
rename from device_tree.c
rename to softmmu/device_tree.c
diff --git a/dma-helpers.c b/softmmu/dma-helpers.c
similarity index 100%
rename from dma-helpers.c
rename to softmmu/dma-helpers.c
diff --git a/softmmu/meson.build b/softmmu/meson.build
index 36c96e7b15..862ab24878 100644
--- a/softmmu/meson.build
+++ b/softmmu/meson.build
@@ -14,3 +14,13 @@ specific_ss.add(when: 'CONFIG_SOFTMMU', if_true: [files(
 specific_ss.add(when: ['CONFIG_SOFTMMU', 'CONFIG_TCG'], if_true: [files(
   'icount.c'
 )])
+
+softmmu_ss.add(files(
+  'bootdevice.c',
+  'dma-helpers.c',
+  'qdev-monitor.c',
+), sdl)
+
+softmmu_ss.add(when: 'CONFIG_TPM', if_true: files('tpm.c'))
+softmmu_ss.add(when: 'CONFIG_SECCOMP', if_true: [files('qemu-seccomp.c'), 
seccomp])
+softmmu_ss.add(when: fdt, if_true: files('device_tree.c'))
diff --git a/qdev-monitor.c b/softmmu/qdev-monitor.c
similarity index 100%
rename from qdev-monitor.c
rename to softmmu/qdev-monitor.c
diff --git a/qemu-seccomp.c b/softmmu/qemu-seccomp.c
similarity index 100%
rename from qemu-seccomp.c
rename to softmmu/qemu-seccomp.c
diff --git a/tpm.c b/softmmu/tpm.c
similarity index 100%
rename from tpm.c
rename to softmmu/tpm.c
-- 
2.26.2

[PULL 29/39] qdev: add "check if address free" callback for buses

[Paolo Bonzini]

Check if an address is free on the bus before plugging in the
device.  This makes it possible to do the check without any
side effects, and to detect the problem early without having
to do it in the realize callback.

Signed-off-by: Paolo Bonzini 
Message-Id: <20201006123904.610658-5-mlevi...@redhat.com>
Signed-off-by: Paolo Bonzini 
---
 hw/core/qdev.c | 17 +++--
 hw/net/virtio-net.c|  2 +-
 hw/sd/core.c   |  3 ++-
 include/hw/qdev-core.h | 13 -
 4 files changed, 30 insertions(+), 5 deletions(-)

diff --git a/hw/core/qdev.c b/hw/core/qdev.c
index 96772a15bd..74db78df36 100644
--- a/hw/core/qdev.c
+++ b/hw/core/qdev.c
@@ -94,13 +94,23 @@ static void bus_add_child(BusState *bus, DeviceState *child)
  0);
 }
 
-void qdev_set_parent_bus(DeviceState *dev, BusState *bus)
+static bool bus_check_address(BusState *bus, DeviceState *child, Error **errp)
+{
+BusClass *bc = BUS_GET_CLASS(bus);
+return !bc->check_address || bc->check_address(bus, child, errp);
+}
+
+bool qdev_set_parent_bus(DeviceState *dev, BusState *bus, Error **errp)
 {
 BusState *old_parent_bus = dev->parent_bus;
 DeviceClass *dc = DEVICE_GET_CLASS(dev);
 
 assert(dc->bus_type && object_dynamic_cast(OBJECT(bus), dc->bus_type));
 
+if (!bus_check_address(bus, dev, errp)) {
+return false;
+}
+
 if (old_parent_bus) {
 trace_qdev_update_parent_bus(dev, object_get_typename(OBJECT(dev)),
 old_parent_bus, object_get_typename(OBJECT(old_parent_bus)),
@@ -126,6 +136,7 @@ void qdev_set_parent_bus(DeviceState *dev, BusState *bus)
 object_unref(OBJECT(old_parent_bus));
 object_unref(OBJECT(dev));
 }
+return true;
 }
 
 DeviceState *qdev_new(const char *name)
@@ -371,7 +382,9 @@ bool qdev_realize(DeviceState *dev, BusState *bus, Error 
**errp)
 assert(!dev->realized && !dev->parent_bus);
 
 if (bus) {
-qdev_set_parent_bus(dev, bus);
+if (!qdev_set_parent_bus(dev, bus, errp)) {
+return false;
+}
 } else {
 assert(!DEVICE_GET_CLASS(dev)->bus_type);
 }
diff --git a/hw/net/virtio-net.c b/hw/net/virtio-net.c
index a160a9da9c..277289d56e 100644
--- a/hw/net/virtio-net.c
+++ b/hw/net/virtio-net.c
@@ -3138,7 +3138,7 @@ static bool failover_replug_primary(VirtIONet *n, Error 
**errp)
 error_setg(errp, "virtio_net: couldn't find primary bus");
 return false;
 }
-qdev_set_parent_bus(n->primary_dev, n->primary_bus);
+qdev_set_parent_bus(n->primary_dev, n->primary_bus, &error_abort);
 n->primary_should_be_hidden = false;
 if (!qemu_opt_set_bool(n->primary_device_opts,
"partially_hotplugged", true, errp)) {
diff --git a/hw/sd/core.c b/hw/sd/core.c
index 957d116f1a..08c93b5903 100644
--- a/hw/sd/core.c
+++ b/hw/sd/core.c
@@ -23,6 +23,7 @@
 #include "hw/qdev-core.h"
 #include "hw/sd/sd.h"
 #include "qemu/module.h"
+#include "qapi/error.h"
 #include "trace.h"
 
 static inline const char *sdbus_name(SDBus *sdbus)
@@ -240,7 +241,7 @@ void sdbus_reparent_card(SDBus *from, SDBus *to)
 readonly = sc->get_readonly(card);
 
 sdbus_set_inserted(from, false);
-qdev_set_parent_bus(DEVICE(card), &to->qbus);
+qdev_set_parent_bus(DEVICE(card), &to->qbus, &error_abort);
 sdbus_set_inserted(to, true);
 sdbus_set_readonly(to, readonly);
 }
diff --git a/include/hw/qdev-core.h b/include/hw/qdev-core.h
index 72064f4dd4..14d476c587 100644
--- a/include/hw/qdev-core.h
+++ b/include/hw/qdev-core.h
@@ -210,13 +210,24 @@ struct BusClass {
 /* FIXME first arg should be BusState */
 void (*print_dev)(Monitor *mon, DeviceState *dev, int indent);
 char *(*get_dev_path)(DeviceState *dev);
+
 /*
  * This callback is used to create Open Firmware device path in accordance
  * with OF spec http://forthworks.com/standards/of1275.pdf. Individual bus
  * bindings can be found at http://playground.sun.com/1275/bindings/.
  */
 char *(*get_fw_dev_path)(DeviceState *dev);
+
 void (*reset)(BusState *bus);
+
+/*
+ * Return whether the device can be added to @bus,
+ * based on the address that was set (via device properties)
+ * before realize.  If not, on return @errp contains the
+ * human-readable error message.
+ */
+bool (*check_address)(BusState *bus, DeviceState *dev, Error **errp);
+
 BusRealize realize;
 BusUnrealize unrealize;
 
@@ -788,7 +799,7 @@ const char *qdev_fw_name(DeviceState *dev);
 Object *qdev_get_machine(void);
 
 /* FIXME: make this a link<> */
-void qdev_set_parent_bus(DeviceState *dev, BusState *bus);
+bool qdev_set_parent_bus(DeviceState *dev, BusState *bus, Error **errp);
 
 extern bool qdev_hotplug;
 extern bool qdev_hot_removed;
-- 
2.26.2

[PULL 14/39] configure: fix performance regression due to PIC objects

[Paolo Bonzini]

Because most files in QEMU are grouped into static libraries, Meson 
conservatively
compiles them with -fPIC.  This is overkill and produces slowdowns up to 20% on
some TCG tests.

As a stopgap measure, use the b_staticpic option to limit the slowdown to
--enable-pie.  https://github.com/mesonbuild/meson/pull/7760 will allow
us to use b_staticpic=false and let Meson do the right thing.

Reported-by: Ahmed Karaman 
Signed-off-by: Paolo Bonzini 
Message-Id: <20200919155639.1045857-1-pbonz...@redhat.com>
Signed-off-by: Paolo Bonzini 
---
 configure | 1 +
 1 file changed, 1 insertion(+)

diff --git a/configure b/configure
index 28df227db5..b553288c5e 100755
--- a/configure
+++ b/configure
@@ -7209,6 +7209,7 @@ NINJA=${ninja:-$PWD/ninjatool} $meson setup \
 -Dwerror=$(if test "$werror" = yes; then echo true; else echo false; 
fi) \
 -Dstrip=$(if test "$strip_opt" = yes; then echo true; else echo false; 
fi) \
 -Db_pie=$(if test "$pie" = yes; then echo true; else echo false; fi) \
+-Db_staticpic=$(if test "$pie" = yes; then echo true; else echo false; 
fi) \
 -Db_coverage=$(if test "$gcov" = yes; then echo true; else echo false; 
fi) \
-Dmalloc=$malloc -Dmalloc_trim=$malloc_trim -Dsparse=$sparse \
-Dkvm=$kvm -Dhax=$hax -Dwhpx=$whpx -Dhvf=$hvf \
-- 
2.26.2

[PULL 07/39] crypto: Move the creation of the library to the main meson.build

[Paolo Bonzini]

From: Philippe Mathieu-Daudé 

Be consistent creating all the libraries in the main meson.build file.

Suggested-by: Paolo Bonzini 
Signed-off-by: Philippe Mathieu-Daudé 
Message-Id: <20201006125602.2311423-8-phi...@redhat.com>
Signed-off-by: Paolo Bonzini 
---
 crypto/meson.build | 10 --
 meson.build| 10 ++
 2 files changed, 10 insertions(+), 10 deletions(-)

diff --git a/crypto/meson.build b/crypto/meson.build
index f6f5ce1ecd..7f37b5d335 100644
--- a/crypto/meson.build
+++ b/crypto/meson.build
@@ -1,4 +1,3 @@
-crypto_ss = ss.source_set()
 crypto_ss.add(genh)
 crypto_ss.add(files(
   'afsplit.c',
@@ -52,15 +51,6 @@ if 'CONFIG_GNUTLS' in config_host
 endif
 
 
-crypto_ss = crypto_ss.apply(config_host, strict: false)
-libcrypto = static_library('crypto', crypto_ss.sources() + genh,
-   dependencies: [crypto_ss.dependencies()],
-   name_suffix: 'fa',
-   build_by_default: false)
-
-crypto = declare_dependency(link_whole: libcrypto,
-dependencies: [authz, qom])
-
 util_ss.add(files('aes.c'))
 util_ss.add(files('init.c'))
 
diff --git a/meson.build b/meson.build
index 28f31e6022..9c199195bf 100644
--- a/meson.build
+++ b/meson.build
@@ -1187,6 +1187,7 @@ block_ss = ss.source_set()
 bsd_user_ss = ss.source_set()
 chardev_ss = ss.source_set()
 common_ss = ss.source_set()
+crypto_ss = ss.source_set()
 io_ss = ss.source_set()
 linux_user_ss = ss.source_set()
 qmp_ss = ss.source_set()
@@ -1452,6 +1453,15 @@ qemu_syms = custom_target('qemu.syms', output: 
'qemu.syms',
 # Library dependencies #
 
 
+crypto_ss = crypto_ss.apply(config_host, strict: false)
+libcrypto = static_library('crypto', crypto_ss.sources() + genh,
+   dependencies: [crypto_ss.dependencies()],
+   name_suffix: 'fa',
+   build_by_default: false)
+
+crypto = declare_dependency(link_whole: libcrypto,
+dependencies: [authz, qom])
+
 io_ss = io_ss.apply(config_host, strict: false)
 libio = static_library('io', io_ss.sources() + genh,
dependencies: [io_ss.dependencies()],
-- 
2.26.2

[PULL 20/39] build-sys: fix git version from -version

[Paolo Bonzini]

From: Marc-André Lureau 

Typo introduced with the script.

Fixes: 2c273f32d3 ("meson: generate qemu-version.h")
Signed-off-by: Marc-André Lureau 
Reported-by: Laszlo Ersek 
Reviewed-by: Eric Blake 
Tested-by: Laszlo Ersek 
Message-Id: <20200929143654.518157-1-marcandre.lur...@redhat.com>
Signed-off-by: Paolo Bonzini 
---
 scripts/qemu-version.sh | 2 +-
 1 file changed, 1 insertion(+), 1 deletion(-)

diff --git a/scripts/qemu-version.sh b/scripts/qemu-version.sh
index 03128c56a2..3f6e7e6d41 100755
--- a/scripts/qemu-version.sh
+++ b/scripts/qemu-version.sh
@@ -9,7 +9,7 @@ version="$3"
 if [ -z "$pkgversion" ]; then
 cd "$dir"
 if [ -e .git ]; then
-pkgversion=$(git describe --match 'v*' --dirty | echo "")
+pkgversion=$(git describe --match 'v*' --dirty) || :
 fi
 fi
 
-- 
2.26.2

[PULL 13/39] qom: fix objects with improper parent type

[Paolo Bonzini]

Some objects accidentally inherit ObjectClass instead of Object.
They compile silently but may crash after downcasting.

In this patch, we introduce a coccinelle script to find broken
declarations and fix them manually with proper base type.

Signed-off-by: Sergey Nizovtsev 
Signed-off-by: Paolo Bonzini 
---
 MAINTAINERS  |  1 +
 include/hw/acpi/vmgenid.h|  2 +-
 include/hw/misc/vmcoreinfo.h |  2 +-
 include/net/can_host.h   |  2 +-
 scripts/coccinelle/qom-parent-type.cocci | 26 
 5 files changed, 30 insertions(+), 3 deletions(-)
 create mode 100644 scripts/coccinelle/qom-parent-type.cocci

diff --git a/MAINTAINERS b/MAINTAINERS
index 7ef459a33c..fcb2c03c2b 100644
--- a/MAINTAINERS
+++ b/MAINTAINERS
@@ -2462,6 +2462,7 @@ F: include/monitor/qdev.h
 F: include/qom/
 F: qapi/qom.json
 F: qapi/qdev.json
+F: scripts/coccinelle/qom-parent-type.cocci
 F: softmmu/qdev-monitor.c
 F: qom/
 F: tests/check-qom-interface.c
diff --git a/include/hw/acpi/vmgenid.h b/include/hw/acpi/vmgenid.h
index d50fbacb8e..cb4ad37fc5 100644
--- a/include/hw/acpi/vmgenid.h
+++ b/include/hw/acpi/vmgenid.h
@@ -19,7 +19,7 @@
 OBJECT_DECLARE_SIMPLE_TYPE(VmGenIdState, VMGENID)
 
 struct VmGenIdState {
-DeviceClass parent_obj;
+DeviceState parent_obj;
 QemuUUID guid;/* The 128-bit GUID seen by the guest */
 uint8_t vmgenid_addr_le[8];   /* Address of the GUID (little-endian) */
 };
diff --git a/include/hw/misc/vmcoreinfo.h b/include/hw/misc/vmcoreinfo.h
index ebada6617a..0b7b55d400 100644
--- a/include/hw/misc/vmcoreinfo.h
+++ b/include/hw/misc/vmcoreinfo.h
@@ -24,7 +24,7 @@ DECLARE_INSTANCE_CHECKER(VMCoreInfoState, VMCOREINFO,
 typedef struct fw_cfg_vmcoreinfo FWCfgVMCoreInfo;
 
 struct VMCoreInfoState {
-DeviceClass parent_obj;
+DeviceState parent_obj;
 
 bool has_vmcoreinfo;
 FWCfgVMCoreInfo vmcoreinfo;
diff --git a/include/net/can_host.h b/include/net/can_host.h
index 4e3ce3f954..caab71bdda 100644
--- a/include/net/can_host.h
+++ b/include/net/can_host.h
@@ -35,7 +35,7 @@
 OBJECT_DECLARE_TYPE(CanHostState, CanHostClass, CAN_HOST)
 
 struct CanHostState {
-ObjectClass oc;
+Object oc;
 
 CanBusState *bus;
 CanBusClientState bus_client;
diff --git a/scripts/coccinelle/qom-parent-type.cocci 
b/scripts/coccinelle/qom-parent-type.cocci
new file mode 100644
index 00..9afb3edd97
--- /dev/null
+++ b/scripts/coccinelle/qom-parent-type.cocci
@@ -0,0 +1,26 @@
+// Highlight object declarations that don't look like object class but
+// accidentally inherit from it.
+
+@match@
+identifier obj_t, fld;
+type parent_t =~ ".*Class$";
+@@
+struct obj_t {
+parent_t fld;
+...
+};
+
+@script:python filter depends on match@
+obj_t << match.obj_t;
+@@
+is_class_obj = obj_t.endswith('Class')
+cocci.include_match(not is_class_obj)
+
+@replacement depends on filter@
+identifier match.obj_t, match.fld;
+type match.parent_t;
+@@
+struct obj_t {
+*   parent_t fld;
+...
+};
-- 
2.26.2

[PULL 39/39] meson: identify more sections of meson.build

[Paolo Bonzini]

Add more headers that clarify code organization.

Signed-off-by: Paolo Bonzini 
---
 meson.build | 36 +---
 1 file changed, 25 insertions(+), 11 deletions(-)

diff --git a/meson.build b/meson.build
index 0c676af194..652c37cceb 100644
--- a/meson.build
+++ b/meson.build
@@ -614,7 +614,9 @@ if not has_malloc_trim and 
get_option('malloc_trim').enabled()
   endif
 endif
 
-# Create config-host.h
+#
+# config-host.h #
+#
 
 config_host_data.set('CONFIG_COCOA', cocoa.found())
 config_host_data.set('CONFIG_LIBUDEV', libudev.found())
@@ -660,6 +662,10 @@ foreach k, v: config_host
   endif
 endforeach
 
+
+# Target configuration #
+
+
 minikconf = find_program('scripts/minikconf.py')
 config_all = {}
 config_all_devices = {}
@@ -866,7 +872,9 @@ config_all += {
   'CONFIG_ALL': true,
 }
 
-# Submodules
+##
+# Submodules #
+##
 
 capstone = not_found
 capstone_opt = get_option('capstone')
@@ -1105,9 +1113,11 @@ config_host_data.set('CONFIG_CAPSTONE', capstone.found())
 config_host_data.set('CONFIG_FDT', fdt.found())
 config_host_data.set('CONFIG_SLIRP', slirp.found())
 
-genh += configure_file(output: 'config-host.h', configuration: 
config_host_data)
+#
+# Generated sources #
+#
 
-# Generators
+genh += configure_file(output: 'config-host.h', configuration: 
config_host_data)
 
 hxtool = find_program('scripts/hxtool')
 shaderinclude = find_program('scripts/shaderinclude.pl')
@@ -1182,7 +1192,9 @@ sphinx_extn_depends = [ meson.source_root() / 
'docs/sphinx/depfile.py',
 meson.source_root() / 'docs/sphinx/qmp_lexer.py',
 qapi_gen_depends ]
 
-# Collect sourcesets.
+###
+# Collect sources #
+###
 
 authz_ss = ss.source_set()
 blockdev_ss = ss.source_set()
@@ -1320,8 +1332,6 @@ if enable_modules
   modulecommon = declare_dependency(link_whole: libmodulecommon, compile_args: 
'-DBUILD_DSO')
 endif
 
-# Build targets from sourcesets
-
 stub_ss = stub_ss.apply(config_all, strict: false)
 
 util_ss.add_all(trace_ss)
@@ -1409,6 +1419,10 @@ specific_ss.add_all(when: 'CONFIG_LINUX_USER', if_true: 
linux_user_ss)
 subdir('tests/qtest/libqos')
 subdir('tests/qtest/fuzz')
 
+
+# Library dependencies #
+
+
 block_mods = []
 softmmu_mods = []
 foreach d, list : modules
@@ -1443,10 +1457,6 @@ qemu_syms = custom_target('qemu.syms', output: 
'qemu.syms',
  capture: true,
  command: [undefsym, nm, '@INPUT@'])
 
-
-# Library dependencies #
-
-
 qom_ss = qom_ss.apply(config_host, strict: false)
 libqom = static_library('qom', qom_ss.sources() + genh,
 dependencies: [qom_ss.dependencies()],
@@ -1797,6 +1807,10 @@ if host_machine.system() == 'windows'
   alias_target('installer', nsis)
 endif
 
+#
+# Configuration summary #
+#
+
 summary_info = {}
 summary_info += {'Install prefix':config_host['prefix']}
 summary_info += {'BIOS directory':config_host['qemu_datadir']}
-- 
2.26.2

[PULL 17/39] docs/devel/qtest: Include libqtest API reference

[Paolo Bonzini]

From: Eduardo Habkost 

Signed-off-by: Eduardo Habkost 
Acked-by: Thomas Huth 
Message-Id: <20201005205228.697463-4-ehabk...@redhat.com>
Signed-off-by: Paolo Bonzini 
---
 docs/devel/qtest.rst  |  6 ++
 tests/qtest/libqos/libqtest.h | 20 ++--
 2 files changed, 16 insertions(+), 10 deletions(-)

diff --git a/docs/devel/qtest.rst b/docs/devel/qtest.rst
index 3bf9ebee7f..075fe5f7d5 100644
--- a/docs/devel/qtest.rst
+++ b/docs/devel/qtest.rst
@@ -64,3 +64,9 @@ QTest Protocol
 
 .. kernel-doc:: softmmu/qtest.c
:doc: QTest Protocol
+
+
+libqtest API reference
+--
+
+.. kernel-doc:: tests/qtest/libqos/libqtest.h
diff --git a/tests/qtest/libqos/libqtest.h b/tests/qtest/libqos/libqtest.h
index a6ee1654f2..209fcf6973 100644
--- a/tests/qtest/libqos/libqtest.h
+++ b/tests/qtest/libqos/libqtest.h
@@ -24,7 +24,7 @@ typedef struct QTestState QTestState;
 
 /**
  * qtest_initf:
- * @fmt...: Format for creating other arguments to pass to QEMU, formatted
+ * @fmt: Format for creating other arguments to pass to QEMU, formatted
  * like sprintf().
  *
  * Convenience wrapper around qtest_init().
@@ -87,7 +87,7 @@ void qtest_quit(QTestState *s);
  * @s: #QTestState instance to operate on.
  * @fds: array of file descriptors
  * @fds_num: number of elements in @fds
- * @fmt...: QMP message to send to qemu, formatted like
+ * @fmt: QMP message to send to qemu, formatted like
  * qobject_from_jsonf_nofail().  See parse_escape() for what's
  * supported after '%'.
  *
@@ -100,7 +100,7 @@ QDict *qtest_qmp_fds(QTestState *s, int *fds, size_t 
fds_num,
 /**
  * qtest_qmp:
  * @s: #QTestState instance to operate on.
- * @fmt...: QMP message to send to qemu, formatted like
+ * @fmt: QMP message to send to qemu, formatted like
  * qobject_from_jsonf_nofail().  See parse_escape() for what's
  * supported after '%'.
  *
@@ -112,7 +112,7 @@ QDict *qtest_qmp(QTestState *s, const char *fmt, ...)
 /**
  * qtest_qmp_send:
  * @s: #QTestState instance to operate on.
- * @fmt...: QMP message to send to qemu, formatted like
+ * @fmt: QMP message to send to qemu, formatted like
  * qobject_from_jsonf_nofail().  See parse_escape() for what's
  * supported after '%'.
  *
@@ -124,7 +124,7 @@ void qtest_qmp_send(QTestState *s, const char *fmt, ...)
 /**
  * qtest_qmp_send_raw:
  * @s: #QTestState instance to operate on.
- * @fmt...: text to send, formatted like sprintf()
+ * @fmt: text to send, formatted like sprintf()
  *
  * Sends text to the QMP monitor verbatim.  Need not be valid JSON;
  * this is useful for negative tests.
@@ -201,7 +201,7 @@ QDict *qtest_qmp_receive(QTestState *s);
 /**
  * qtest_qmp_eventwait:
  * @s: #QTestState instance to operate on.
- * @s: #event event to wait for.
+ * @event: event to wait for.
  *
  * Continuously polls for QMP responses until it receives the desired event.
  */
@@ -210,7 +210,7 @@ void qtest_qmp_eventwait(QTestState *s, const char *event);
 /**
  * qtest_qmp_eventwait_ref:
  * @s: #QTestState instance to operate on.
- * @s: #event event to wait for.
+ * @event: event to wait for.
  *
  * Continuously polls for QMP responses until it receives the desired event.
  * Returns a copy of the event for further investigation.
@@ -237,7 +237,7 @@ QDict *qtest_qmp_receive_success(QTestState *s,
 /**
  * qtest_hmp:
  * @s: #QTestState instance to operate on.
- * @fmt...: HMP command to send to QEMU, formats arguments like sprintf().
+ * @fmt: HMP command to send to QEMU, formats arguments like sprintf().
  *
  * Send HMP command to QEMU via QMP's human-monitor-command.
  * QMP events are discarded.
@@ -629,7 +629,7 @@ void qtest_add_abrt_handler(GHookFunc fn, const void *data);
 /**
  * qtest_qmp_assert_success:
  * @qts: QTestState instance to operate on
- * @fmt...: QMP message to send to qemu, formatted like
+ * @fmt: QMP message to send to qemu, formatted like
  * qobject_from_jsonf_nofail().  See parse_escape() for what's
  * supported after '%'.
  *
@@ -676,7 +676,7 @@ void qtest_qmp_device_add_qdict(QTestState *qts, const char 
*drv,
  * @qts: QTestState instance to operate on
  * @driver: Name of the device that should be added
  * @id: Identification string
- * @fmt...: QMP message to send to qemu, formatted like
+ * @fmt: QMP message to send to qemu, formatted like
  * qobject_from_jsonf_nofail().  See parse_escape() for what's
  * supported after '%'.
  *
-- 
2.26.2

[PULL 10/39] hw/nvram: Always register FW_CFG_DATA_GENERATOR_INTERFACE

[Paolo Bonzini]

From: Philippe Mathieu-Daudé 

While the FW_CFG_DATA_GENERATOR_INTERFACE is only consumed
by a device only available using system-mode (fw_cfg), it is
implemented by a crypto component (tls-cipher-suites) which
is always available when crypto is used.

Commit 69699f3055 introduced the following error in the
qemu-storage-daemon binary:

  $ echo -e \
'{"execute": "qmp_capabilities"}\r\n{"execute": 
"qom-list-types"}\r\n{"execute": "quit"}\r\n' \
| storage-daemon/qemu-storage-daemon --chardev stdio,id=qmp0  --monitor qmp0
  {"QMP": {"version": {"qemu": {"micro": 50, "minor": 1, "major": 5}, 
"package": ""}, "capabilities": ["oob"]}}
  {"return": {}}
  missing interface 'fw_cfg-data-generator' for object 'tls-creds'
  Aborted (core dumped)

Since QOM dependencies are resolved at runtime, this issue
could not be triggered at linktime, and we don't have test
running the qemu-storage-daemon binary.

Fix by always registering the QOM interface.

Reported-by: Kevin Wolf 
Fixes: 69699f3055 ("crypto/tls-cipher-suites: Produce fw_cfg consumable blob")
Signed-off-by: Philippe Mathieu-Daudé 
Reviewed-by: Laszlo Ersek 
Reviewed-by: Paolo Bonzini 
Message-Id: <20201006111909.2302081-2-phi...@redhat.com>
Signed-off-by: Paolo Bonzini 
---
 MAINTAINERS |  2 +-
 hw/nvram/fw_cfg-interface.c | 23 +++
 hw/nvram/fw_cfg.c   |  7 ---
 hw/nvram/meson.build|  3 +++
 4 files changed, 27 insertions(+), 8 deletions(-)
 create mode 100644 hw/nvram/fw_cfg-interface.c

diff --git a/MAINTAINERS b/MAINTAINERS
index e9d85cc873..8d50b96c33 100644
--- a/MAINTAINERS
+++ b/MAINTAINERS
@@ -2055,7 +2055,7 @@ R: Laszlo Ersek 
 R: Gerd Hoffmann 
 S: Supported
 F: docs/specs/fw_cfg.txt
-F: hw/nvram/fw_cfg.c
+F: hw/nvram/fw_cfg*.c
 F: stubs/fw_cfg.c
 F: include/hw/nvram/fw_cfg.h
 F: include/standard-headers/linux/qemu_fw_cfg.h
diff --git a/hw/nvram/fw_cfg-interface.c b/hw/nvram/fw_cfg-interface.c
new file mode 100644
index 00..0e93feeae5
--- /dev/null
+++ b/hw/nvram/fw_cfg-interface.c
@@ -0,0 +1,23 @@
+/*
+ * QEMU Firmware configuration device emulation (QOM interfaces)
+ *
+ * Copyright 2020 Red Hat, Inc.
+ *
+ * SPDX-License-Identifier: GPL-2.0-or-later
+ */
+
+#include "qemu/osdep.h"
+#include "hw/nvram/fw_cfg.h"
+
+static const TypeInfo fw_cfg_data_generator_interface_info = {
+.parent = TYPE_INTERFACE,
+.name = TYPE_FW_CFG_DATA_GENERATOR_INTERFACE,
+.class_size = sizeof(FWCfgDataGeneratorClass),
+};
+
+static void fw_cfg_register_interfaces(void)
+{
+type_register_static(&fw_cfg_data_generator_interface_info);
+}
+
+type_init(fw_cfg_register_interfaces)
diff --git a/hw/nvram/fw_cfg.c b/hw/nvram/fw_cfg.c
index 0e95d057fd..08539a1aab 100644
--- a/hw/nvram/fw_cfg.c
+++ b/hw/nvram/fw_cfg.c
@@ -1360,18 +1360,11 @@ static const TypeInfo fw_cfg_mem_info = {
 .class_init= fw_cfg_mem_class_init,
 };
 
-static const TypeInfo fw_cfg_data_generator_interface_info = {
-.parent = TYPE_INTERFACE,
-.name = TYPE_FW_CFG_DATA_GENERATOR_INTERFACE,
-.class_size = sizeof(FWCfgDataGeneratorClass),
-};
-
 static void fw_cfg_register_types(void)
 {
 type_register_static(&fw_cfg_info);
 type_register_static(&fw_cfg_io_info);
 type_register_static(&fw_cfg_mem_info);
-type_register_static(&fw_cfg_data_generator_interface_info);
 }
 
 type_init(fw_cfg_register_types)
diff --git a/hw/nvram/meson.build b/hw/nvram/meson.build
index 1f2ed013b2..fd2951a860 100644
--- a/hw/nvram/meson.build
+++ b/hw/nvram/meson.build
@@ -1,3 +1,6 @@
+# QOM interfaces must be available anytime QOM is used.
+qom_ss.add(files('fw_cfg-interface.c'))
+
 softmmu_ss.add(files('fw_cfg.c'))
 softmmu_ss.add(when: 'CONFIG_CHRP_NVRAM', if_true: files('chrp_nvram.c'))
 softmmu_ss.add(when: 'CONFIG_DS1225Y', if_true: files('ds1225y.c'))
-- 
2.26.2

Re: [PATCH v10 6/7] test: new qTest case to test the vhost-user-blk-server

[Coiby Xu]

On Wed, Sep 23, 2020 at 01:36:06PM +0100, Stefan Hajnoczi wrote:

On Fri, Sep 18, 2020 at 04:09:11PM +0800, Coiby Xu wrote:

+int qtest_socket_client(char *server_socket_path)
+{
+struct sockaddr_un serv_addr;
+int sock;
+int ret;
+int retries = 0;
+sock = socket(PF_UNIX, SOCK_STREAM, 0);
+g_assert_cmpint(sock, !=, -1);
+serv_addr.sun_family = AF_UNIX;
+snprintf(serv_addr.sun_path, sizeof(serv_addr.sun_path), "%s",
+ server_socket_path);
+
+for (retries = 0; retries < 3; retries++) {
+ret = connect(sock, (struct sockaddr *)&serv_addr, sizeof(serv_addr));
+if (ret == 0) {
+break;
+}
+g_usleep(G_USEC_PER_SEC);
+}


This is a race condition. On a heavily loaded machine the server might
not be available within 3 seconds and the test will fail randomly.

Solutions:
1. Wait output from the server indicating it is ready (e.g. 'Listening
  on /path/to/foo.sock...') when spawning the server process.
2. Create the listen socket and pass the fd to the server process. This
  way the socket already exists can the client will block in connect
  until the server accepts the connection.
3. Create a socketpair. Pass one side to the server and use the other
  side in the client.

However, I think this is okay for now. After my patch series that
converts the vhost-user-blk server to the new block exports API we can
consider how to pass file descriptors.


+static void quit_storage_daemon(void *qmp_test_state)
+{
+const char quit_str[] = "{ 'execute': 'quit' }";
+
+/* Before quiting storate-daemon, quit qemu to avoid dubious messages */
+qobject_unref(qtest_qmp(global_qtest, quit_str));
+
+/*
+ * Give storage-daemon enough time to wake up&terminate
+ * vu_client_trip coroutine so the Coroutine object could
+ * be cleaned up. Otherwise LeakSanitizer would complain
+ * about memory leaks.
+ */
+g_usleep(1000);


Also a race that may cause random failures. This can be addressed after
the block exports API conversion.


Thank you for spotting two race conditions!

--
Best regards,
Coiby

[PULL 26/39] qtest: switch users back to qtest_qmp_receive

[Paolo Bonzini]

From: Maxim Levitsky 

Let test use the new functionality for buffering events.
The only remaining users of qtest_qmp_receive_dict are tests
that fuzz the QMP protocol.

Tested with 'make check-qtest'.

Signed-off-by: Maxim Levitsky 
Message-Id: <20201006123904.610658-4-mlevi...@redhat.com>
Signed-off-by: Paolo Bonzini 
---
 tests/qtest/ahci-test.c  |  4 ++--
 tests/qtest/drive_del-test.c |  9 +++--
 tests/qtest/libqtest.c   | 12 +++-
 tests/qtest/pvpanic-test.c   |  4 +---
 tests/qtest/tpm-util.c   |  8 ++--
 5 files changed, 15 insertions(+), 22 deletions(-)

diff --git a/tests/qtest/ahci-test.c b/tests/qtest/ahci-test.c
index d42ebaeb4c..5e1954852e 100644
--- a/tests/qtest/ahci-test.c
+++ b/tests/qtest/ahci-test.c
@@ -1590,7 +1590,7 @@ static void test_atapi_tray(void)
 qtest_qmp_send(ahci->parent->qts, "{'execute': 'blockdev-open-tray', "
 "'arguments': {'id': 'cd0'}}");
 atapi_wait_tray(ahci, true);
-rsp = qtest_qmp_receive_dict(ahci->parent->qts);
+rsp = qtest_qmp_receive(ahci->parent->qts);
 qobject_unref(rsp);
 
 qmp_discard_response(ahci->parent->qts,
@@ -1620,7 +1620,7 @@ static void test_atapi_tray(void)
 qtest_qmp_send(ahci->parent->qts, "{'execute': 'blockdev-close-tray', "
"'arguments': {'id': 'cd0'}}");
 atapi_wait_tray(ahci, false);
-rsp = qtest_qmp_receive_dict(ahci->parent->qts);
+rsp = qtest_qmp_receive(ahci->parent->qts);
 qobject_unref(rsp);
 
 /* Now, to convince ATAPI we understand the media has changed... */
diff --git a/tests/qtest/drive_del-test.c b/tests/qtest/drive_del-test.c
index ba0cd77445..9d20a1ed8b 100644
--- a/tests/qtest/drive_del-test.c
+++ b/tests/qtest/drive_del-test.c
@@ -15,9 +15,6 @@
 #include "libqos/virtio.h"
 #include "qapi/qmp/qdict.h"
 
-/* TODO actually test the results and get rid of this */
-#define qmp_discard_response(q, ...) qobject_unref(qtest_qmp(q, __VA_ARGS__))
-
 static void drive_add(QTestState *qts)
 {
 char *resp = qtest_hmp(qts, "drive_add 0 if=none,id=drive0");
@@ -38,13 +35,13 @@ static void device_del(QTestState *qts)
 {
 QDict *response;
 
-/* Complication: ignore DEVICE_DELETED event */
-qmp_discard_response(qts, "{'execute': 'device_del',"
+response = qtest_qmp(qts, "{'execute': 'device_del',"
  " 'arguments': { 'id': 'dev0' } }");
-response = qtest_qmp_receive_dict(qts);
 g_assert(response);
 g_assert(qdict_haskey(response, "return"));
 qobject_unref(response);
+
+qtest_qmp_eventwait(qts, "DEVICE_DELETED");
 }
 
 static void test_drive_without_dev(void)
diff --git a/tests/qtest/libqtest.c b/tests/qtest/libqtest.c
index baac667b8d..08929f5ff6 100644
--- a/tests/qtest/libqtest.c
+++ b/tests/qtest/libqtest.c
@@ -324,7 +324,7 @@ QTestState *qtest_init(const char *extra_args)
 QDict *greeting;
 
 /* Read the QMP greeting and then do the handshake */
-greeting = qtest_qmp_receive_dict(s);
+greeting = qtest_qmp_receive(s);
 qobject_unref(greeting);
 qobject_unref(qtest_qmp(s, "{ 'execute': 'qmp_capabilities' }"));
 
@@ -700,7 +700,7 @@ QDict *qtest_vqmp_fds(QTestState *s, int *fds, size_t 
fds_num,
 qtest_qmp_vsend_fds(s, fds, fds_num, fmt, ap);
 
 /* Receive reply */
-return qtest_qmp_receive_dict(s);
+return qtest_qmp_receive(s);
 }
 
 QDict *qtest_vqmp(QTestState *s, const char *fmt, va_list ap)
@@ -708,7 +708,7 @@ QDict *qtest_vqmp(QTestState *s, const char *fmt, va_list 
ap)
 qtest_qmp_vsend(s, fmt, ap);
 
 /* Receive reply */
-return qtest_qmp_receive_dict(s);
+return qtest_qmp_receive(s);
 }
 
 QDict *qmp_fd(int fd, const char *fmt, ...)
@@ -850,12 +850,6 @@ char *qtest_vhmp(QTestState *s, const char *fmt, va_list 
ap)
  " 'arguments': {'command-line': %s}}",
  cmd);
 ret = g_strdup(qdict_get_try_str(resp, "return"));
-while (ret == NULL && qdict_get_try_str(resp, "event")) {
-/* Ignore asynchronous QMP events */
-qobject_unref(resp);
-resp = qtest_qmp_receive_dict(s);
-ret = g_strdup(qdict_get_try_str(resp, "return"));
-}
 g_assert(ret);
 qobject_unref(resp);
 g_free(cmd);
diff --git a/tests/qtest/pvpanic-test.c b/tests/qtest/pvpanic-test.c
index b0b20ad340..0657de797f 100644
--- a/tests/qtest/pvpanic-test.c
+++ b/tests/qtest/pvpanic-test.c
@@ -24,9 +24,7 @@ static void test_panic(void)
 
 qtest_outb(qts, 0x505, 0x1);
 
-response = qtest_qmp_receive_dict(qts);
-g_assert(qdict_haskey(response, "event"));
-g_assert_cmpstr(qdict_get_str(response, "event"), ==, "GUEST_PANICKED");
+response = qtest_qmp_eventwait_ref(qts, "GUEST_PANICKED");
 g_assert(qdict_haskey(response, "data"));
 data = qdict_get_qdict(response, "data");
 g_assert(qdict_haskey(data, "action"));
diff --git a/tests/qtest/tpm-util.c b/tests/qtest/tpm-util.c
index 3ed6c8548a..5a33a6ef0f 100644
--- a/tests/qtest/tpm-util

[PULL 12/39] exec: split out non-softmmu-specific parts

[Paolo Bonzini]

Over the years, most parts of exec.c that were not specific to softmmu
have been moved to accel/tcg; what's left is mostly the low-level part
of the memory API, which includes RAMBlock and AddressSpaceDispatch.
However exec.c also hosts 4-500 lines of code for the target specific
parts of the CPU QOM object, plus a few functions for user-mode
emulation that do not have a better place (they are not TCG-specific so
accel/tcg/user-exec.c is not a good place either).

Move these parts to a new file, so that exec.c can be moved to
softmmu/physmem.c.

Signed-off-by: Paolo Bonzini 
---
 MAINTAINERS |   4 +-
 cpu.c   | 452 +++
 include/exec/cpu-common.h   |   3 +
 meson.build |   2 +-
 softmmu/meson.build |   3 +-
 exec.c => softmmu/physmem.c | 454 +---
 6 files changed, 467 insertions(+), 451 deletions(-)
 create mode 100644 cpu.c
 rename exec.c => softmmu/physmem.c (91%)

diff --git a/MAINTAINERS b/MAINTAINERS
index dda54f000d..7ef459a33c 100644
--- a/MAINTAINERS
+++ b/MAINTAINERS
@@ -117,7 +117,6 @@ R: Paolo Bonzini 
 S: Maintained
 F: softmmu/cpus.c
 F: cpus-common.c
-F: exec.c
 F: accel/tcg/
 F: accel/stubs/tcg-stub.c
 F: scripts/decodetree.py
@@ -1525,6 +1524,7 @@ Machine core
 M: Eduardo Habkost 
 M: Marcel Apfelbaum 
 S: Supported
+F: cpu.c
 F: hw/core/cpu.c
 F: hw/core/machine-qmp-cmds.c
 F: hw/core/machine.c
@@ -2284,8 +2284,8 @@ F: include/exec/ramblock.h
 F: softmmu/dma-helpers.c
 F: softmmu/ioport.c
 F: softmmu/memory.c
+F: softmmu/physmem.c
 F: include/exec/memory-internal.h
-F: exec.c
 F: scripts/coccinelle/memory-region-housekeeping.cocci
 
 SPICE
diff --git a/cpu.c b/cpu.c
new file mode 100644
index 00..0be5dcb6f3
--- /dev/null
+++ b/cpu.c
@@ -0,0 +1,452 @@
+/*
+ * Target-specific parts of the CPU object
+ *
+ *  Copyright (c) 2003 Fabrice Bellard
+ *
+ * This library is free software; you can redistribute it and/or
+ * modify it under the terms of the GNU Lesser General Public
+ * License as published by the Free Software Foundation; either
+ * version 2 of the License, or (at your option) any later version.
+ *
+ * This library is distributed in the hope that it will be useful,
+ * but WITHOUT ANY WARRANTY; without even the implied warranty of
+ * MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the GNU
+ * Lesser General Public License for more details.
+ *
+ * You should have received a copy of the GNU Lesser General Public
+ * License along with this library; if not, see .
+ */
+
+#include "qemu/osdep.h"
+#include "qemu-common.h"
+#include "qapi/error.h"
+
+#include "exec/target_page.h"
+#include "hw/qdev-core.h"
+#include "hw/qdev-properties.h"
+#include "qemu/error-report.h"
+#include "migration/vmstate.h"
+#ifdef CONFIG_USER_ONLY
+#include "qemu.h"
+#else
+#include "exec/address-spaces.h"
+#endif
+#include "sysemu/tcg.h"
+#include "sysemu/kvm.h"
+#include "sysemu/replay.h"
+#include "translate-all.h"
+#include "exec/log.h"
+
+uintptr_t qemu_host_page_size;
+intptr_t qemu_host_page_mask;
+
+#ifndef CONFIG_USER_ONLY
+static int cpu_common_post_load(void *opaque, int version_id)
+{
+CPUState *cpu = opaque;
+
+/* 0x01 was CPU_INTERRUPT_EXIT. This line can be removed when the
+   version_id is increased. */
+cpu->interrupt_request &= ~0x01;
+tlb_flush(cpu);
+
+/* loadvm has just updated the content of RAM, bypassing the
+ * usual mechanisms that ensure we flush TBs for writes to
+ * memory we've translated code from. So we must flush all TBs,
+ * which will now be stale.
+ */
+tb_flush(cpu);
+
+return 0;
+}
+
+static int cpu_common_pre_load(void *opaque)
+{
+CPUState *cpu = opaque;
+
+cpu->exception_index = -1;
+
+return 0;
+}
+
+static bool cpu_common_exception_index_needed(void *opaque)
+{
+CPUState *cpu = opaque;
+
+return tcg_enabled() && cpu->exception_index != -1;
+}
+
+static const VMStateDescription vmstate_cpu_common_exception_index = {
+.name = "cpu_common/exception_index",
+.version_id = 1,
+.minimum_version_id = 1,
+.needed = cpu_common_exception_index_needed,
+.fields = (VMStateField[]) {
+VMSTATE_INT32(exception_index, CPUState),
+VMSTATE_END_OF_LIST()
+}
+};
+
+static bool cpu_common_crash_occurred_needed(void *opaque)
+{
+CPUState *cpu = opaque;
+
+return cpu->crash_occurred;
+}
+
+static const VMStateDescription vmstate_cpu_common_crash_occurred = {
+.name = "cpu_common/crash_occurred",
+.version_id = 1,
+.minimum_version_id = 1,
+.needed = cpu_common_crash_occurred_needed,
+.fields = (VMStateField[]) {
+VMSTATE_BOOL(crash_occurred, CPUState),
+VMSTATE_END_OF_LIST()
+}
+};
+
+const VMStateDescription vmstate_cpu_common = {
+.name = "cpu_common",
+.version_id = 1,
+.minimum_version_id = 1,
+.pre_load = cpu_common_pre_load,
+.post_load = c

[PATCH V2 0/5] Support RISC-V migration

[Yifei Jiang]

This patches supported RISC-V migration based on tcg accel. And we have
verified related migration features such as snapshot and live migration.

A few weeks ago, we submitted RFC patches about supporting RISC-V migration
based on kvm accel: https://www.spinics.net/lists/kvm/msg223605.html.
And we found that tcg accelerated migration can be supported with a few
changes. Most of the devices have already implemented the migration
interface, so, to achieve the tcg accelerated migration, we just need to
add vmstate of both cpu and sifive_plic.

Changes since v1:
1. Add license head to target/riscv/machine.c.
2. Regenerate some state of PMP at post_load hook.

Yifei Jiang (5):
  target/riscv: Add basic vmstate description of CPU
  target/riscv: Add PMP state description
  target/riscv: Add H extension state description
  target/riscv: Add V extension state description
  target/riscv: Add sifive_plic vmstate

 hw/intc/sifive_plic.c|  26 -
 hw/intc/sifive_plic.h|   1 +
 target/riscv/cpu.c   |   7 --
 target/riscv/cpu.h   |   4 +
 target/riscv/machine.c   | 203 +++
 target/riscv/meson.build |   3 +-
 target/riscv/pmp.c   |  29 +++---
 target/riscv/pmp.h   |   2 +
 8 files changed, 255 insertions(+), 20 deletions(-)
 create mode 100644 target/riscv/machine.c

-- 
2.19.1

[PULL 30/39] scsi/scsi_bus: switch search direction in scsi_device_find

[Paolo Bonzini]

From: Maxim Levitsky 

This change will allow us to convert the bus children list to RCU,
while not changing the logic of this function

Signed-off-by: Maxim Levitsky 
Reviewed-by: Stefan Hajnoczi 
Message-Id: <20200913160259.32145-2-mlevi...@redhat.com>
Signed-off-by: Paolo Bonzini 
---
 hw/scsi/scsi-bus.c | 12 ++--
 1 file changed, 10 insertions(+), 2 deletions(-)

diff --git a/hw/scsi/scsi-bus.c b/hw/scsi/scsi-bus.c
index 3284a5d1fb..6b1ed7ae9a 100644
--- a/hw/scsi/scsi-bus.c
+++ b/hw/scsi/scsi-bus.c
@@ -1572,7 +1572,7 @@ SCSIDevice *scsi_device_find(SCSIBus *bus, int channel, 
int id, int lun)
 BusChild *kid;
 SCSIDevice *target_dev = NULL;
 
-QTAILQ_FOREACH_REVERSE(kid, &bus->qbus.children, sibling) {
+QTAILQ_FOREACH(kid, &bus->qbus.children, sibling) {
 DeviceState *qdev = kid->child;
 SCSIDevice *dev = SCSI_DEVICE(qdev);
 
@@ -1580,7 +1580,15 @@ SCSIDevice *scsi_device_find(SCSIBus *bus, int channel, 
int id, int lun)
 if (dev->lun == lun) {
 return dev;
 }
-target_dev = dev;
+
+/*
+ * If we don't find exact match (channel/bus/lun),
+ * we will return the first device which matches channel/bus
+ */
+
+if (!target_dev) {
+target_dev = dev;
+}
 }
 }
 return target_dev;
-- 
2.26.2

[PULL 08/39] authz: Move the creation of the library to the main meson.build

[Paolo Bonzini]

From: Philippe Mathieu-Daudé 

Be consistent creating all the libraries in the main meson.build file.

Suggested-by: Paolo Bonzini 
Signed-off-by: Philippe Mathieu-Daudé 
Message-Id: <20201006125602.2311423-9-phi...@redhat.com>
Signed-off-by: Paolo Bonzini 
---
 authz/meson.build | 10 --
 meson.build   | 10 ++
 2 files changed, 10 insertions(+), 10 deletions(-)

diff --git a/authz/meson.build b/authz/meson.build
index 516d71f2e2..88fa7769cb 100644
--- a/authz/meson.build
+++ b/authz/meson.build
@@ -1,4 +1,3 @@
-authz_ss = ss.source_set()
 authz_ss.add(genh)
 authz_ss.add(files(
   'base.c',
@@ -8,12 +7,3 @@ authz_ss.add(files(
 ))
 
 authz_ss.add(when: ['CONFIG_AUTH_PAM', pam], if_true: files('pamacct.c'))
-
-authz_ss = authz_ss.apply(config_host, strict: false)
-libauthz = static_library('authz', authz_ss.sources() + genh,
-  dependencies: [authz_ss.dependencies()],
-  name_suffix: 'fa',
-  build_by_default: false)
-
-authz = declare_dependency(link_whole: libauthz,
-   dependencies: qom)
diff --git a/meson.build b/meson.build
index 9c199195bf..2736f74b8f 100644
--- a/meson.build
+++ b/meson.build
@@ -1182,6 +1182,7 @@ sphinx_extn_depends = [ meson.source_root() / 
'docs/sphinx/depfile.py',
 
 # Collect sourcesets.
 
+authz_ss = ss.source_set()
 blockdev_ss = ss.source_set()
 block_ss = ss.source_set()
 bsd_user_ss = ss.source_set()
@@ -1453,6 +1454,15 @@ qemu_syms = custom_target('qemu.syms', output: 
'qemu.syms',
 # Library dependencies #
 
 
+authz_ss = authz_ss.apply(config_host, strict: false)
+libauthz = static_library('authz', authz_ss.sources() + genh,
+  dependencies: [authz_ss.dependencies()],
+  name_suffix: 'fa',
+  build_by_default: false)
+
+authz = declare_dependency(link_whole: libauthz,
+   dependencies: qom)
+
 crypto_ss = crypto_ss.apply(config_host, strict: false)
 libcrypto = static_library('crypto', crypto_ss.sources() + genh,
dependencies: [crypto_ss.dependencies()],
-- 
2.26.2

[PULL 22/39] qtest: rename qtest_qmp_receive to qtest_qmp_receive_dict

[Paolo Bonzini]

From: Maxim Levitsky 

In the next patch a new version of qtest_qmp_receive will be
reintroduced that will buffer received qmp events for later
consumption in qtest_qmp_eventwait_ref

No functional change intended.

Suggested-by: Paolo Bonzini 
Signed-off-by: Maxim Levitsky 
Signed-off-by: Paolo Bonzini 
---
 tests/qtest/ahci-test.c|  4 ++--
 tests/qtest/device-plug-test.c |  2 +-
 tests/qtest/drive_del-test.c   |  2 +-
 tests/qtest/libqos/libqtest.h  |  4 ++--
 tests/qtest/libqtest.c | 16 
 tests/qtest/pvpanic-test.c |  2 +-
 tests/qtest/qmp-test.c | 18 +-
 7 files changed, 24 insertions(+), 24 deletions(-)

diff --git a/tests/qtest/ahci-test.c b/tests/qtest/ahci-test.c
index 5e1954852e..d42ebaeb4c 100644
--- a/tests/qtest/ahci-test.c
+++ b/tests/qtest/ahci-test.c
@@ -1590,7 +1590,7 @@ static void test_atapi_tray(void)
 qtest_qmp_send(ahci->parent->qts, "{'execute': 'blockdev-open-tray', "
 "'arguments': {'id': 'cd0'}}");
 atapi_wait_tray(ahci, true);
-rsp = qtest_qmp_receive(ahci->parent->qts);
+rsp = qtest_qmp_receive_dict(ahci->parent->qts);
 qobject_unref(rsp);
 
 qmp_discard_response(ahci->parent->qts,
@@ -1620,7 +1620,7 @@ static void test_atapi_tray(void)
 qtest_qmp_send(ahci->parent->qts, "{'execute': 'blockdev-close-tray', "
"'arguments': {'id': 'cd0'}}");
 atapi_wait_tray(ahci, false);
-rsp = qtest_qmp_receive(ahci->parent->qts);
+rsp = qtest_qmp_receive_dict(ahci->parent->qts);
 qobject_unref(rsp);
 
 /* Now, to convince ATAPI we understand the media has changed... */
diff --git a/tests/qtest/device-plug-test.c b/tests/qtest/device-plug-test.c
index 9214892741..a2247856be 100644
--- a/tests/qtest/device-plug-test.c
+++ b/tests/qtest/device-plug-test.c
@@ -23,7 +23,7 @@ static void device_del_start(QTestState *qtest, const char 
*id)
 
 static void device_del_finish(QTestState *qtest)
 {
-QDict *resp = qtest_qmp_receive(qtest);
+QDict *resp = qtest_qmp_receive_dict(qtest);
 
 g_assert(qdict_haskey(resp, "return"));
 qobject_unref(resp);
diff --git a/tests/qtest/drive_del-test.c b/tests/qtest/drive_del-test.c
index 2d765865ce..ba0cd77445 100644
--- a/tests/qtest/drive_del-test.c
+++ b/tests/qtest/drive_del-test.c
@@ -41,7 +41,7 @@ static void device_del(QTestState *qts)
 /* Complication: ignore DEVICE_DELETED event */
 qmp_discard_response(qts, "{'execute': 'device_del',"
  " 'arguments': { 'id': 'dev0' } }");
-response = qtest_qmp_receive(qts);
+response = qtest_qmp_receive_dict(qts);
 g_assert(response);
 g_assert(qdict_haskey(response, "return"));
 qobject_unref(response);
diff --git a/tests/qtest/libqos/libqtest.h b/tests/qtest/libqos/libqtest.h
index 209fcf6973..9b3f99b322 100644
--- a/tests/qtest/libqos/libqtest.h
+++ b/tests/qtest/libqos/libqtest.h
@@ -191,12 +191,12 @@ void qtest_qmp_vsend(QTestState *s, const char *fmt, 
va_list ap)
 GCC_FMT_ATTR(2, 0);
 
 /**
- * qtest_receive:
+ * qtest_qmp_receive_dict:
  * @s: #QTestState instance to operate on.
  *
  * Reads a QMP message from QEMU and returns the response.
  */
-QDict *qtest_qmp_receive(QTestState *s);
+QDict *qtest_qmp_receive_dict(QTestState *s);
 
 /**
  * qtest_qmp_eventwait:
diff --git a/tests/qtest/libqtest.c b/tests/qtest/libqtest.c
index 58f58e1ece..dadc465825 100644
--- a/tests/qtest/libqtest.c
+++ b/tests/qtest/libqtest.c
@@ -322,7 +322,7 @@ QTestState *qtest_init(const char *extra_args)
 QDict *greeting;
 
 /* Read the QMP greeting and then do the handshake */
-greeting = qtest_qmp_receive(s);
+greeting = qtest_qmp_receive_dict(s);
 qobject_unref(greeting);
 qobject_unref(qtest_qmp(s, "{ 'execute': 'qmp_capabilities' }"));
 
@@ -603,7 +603,7 @@ QDict *qmp_fd_receive(int fd)
 return qmp.response;
 }
 
-QDict *qtest_qmp_receive(QTestState *s)
+QDict *qtest_qmp_receive_dict(QTestState *s)
 {
 return qmp_fd_receive(s->qmp_fd);
 }
@@ -678,7 +678,7 @@ QDict *qtest_vqmp_fds(QTestState *s, int *fds, size_t 
fds_num,
 qtest_qmp_vsend_fds(s, fds, fds_num, fmt, ap);
 
 /* Receive reply */
-return qtest_qmp_receive(s);
+return qtest_qmp_receive_dict(s);
 }
 
 QDict *qtest_vqmp(QTestState *s, const char *fmt, va_list ap)
@@ -686,7 +686,7 @@ QDict *qtest_vqmp(QTestState *s, const char *fmt, va_list 
ap)
 qtest_qmp_vsend(s, fmt, ap);
 
 /* Receive reply */
-return qtest_qmp_receive(s);
+return qtest_qmp_receive_dict(s);
 }
 
 QDict *qmp_fd(int fd, const char *fmt, ...)
@@ -776,7 +776,7 @@ QDict *qtest_qmp_eventwait_ref(QTestState *s, const char 
*event)
 QDict *response;
 
 for (;;) {
-response = qtest_qmp_receive(s);
+response = qtest_qmp_receive_dict(s);
 if ((qdict_haskey(response, "event")) &&
 (strcmp(qdict_get_str(response, "event"), event) == 0)) {
 return response;
@@ -807,7 +807,7 @@ char *qte

[PATCH V2 1/5] target/riscv: Add basic vmstate description of CPU

[Yifei Jiang]

Add basic CPU state description to the newly created machine.c

Signed-off-by: Yifei Jiang 
Signed-off-by: Yipeng Yin 
---
 target/riscv/cpu.c   |  7 
 target/riscv/cpu.h   |  4 +++
 target/riscv/machine.c   | 77 
 target/riscv/meson.build |  3 +-
 4 files changed, 83 insertions(+), 8 deletions(-)
 create mode 100644 target/riscv/machine.c

diff --git a/target/riscv/cpu.c b/target/riscv/cpu.c
index 0bbfd7f457..bf396e2916 100644
--- a/target/riscv/cpu.c
+++ b/target/riscv/cpu.c
@@ -496,13 +496,6 @@ static void riscv_cpu_init(Object *obj)
 cpu_set_cpustate_pointers(cpu);
 }
 
-#ifndef CONFIG_USER_ONLY
-static const VMStateDescription vmstate_riscv_cpu = {
-.name = "cpu",
-.unmigratable = 1,
-};
-#endif
-
 static Property riscv_cpu_properties[] = {
 DEFINE_PROP_BOOL("i", RISCVCPU, cfg.ext_i, true),
 DEFINE_PROP_BOOL("e", RISCVCPU, cfg.ext_e, false),
diff --git a/target/riscv/cpu.h b/target/riscv/cpu.h
index de275782e6..8440ea0793 100644
--- a/target/riscv/cpu.h
+++ b/target/riscv/cpu.h
@@ -311,6 +311,10 @@ extern const char * const riscv_fpr_regnames[];
 extern const char * const riscv_excp_names[];
 extern const char * const riscv_intr_names[];
 
+#ifndef CONFIG_USER_ONLY
+extern const VMStateDescription vmstate_riscv_cpu;
+#endif
+
 const char *riscv_cpu_get_trap_name(target_ulong cause, bool async);
 void riscv_cpu_do_interrupt(CPUState *cpu);
 int riscv_cpu_gdb_read_register(CPUState *cpu, GByteArray *buf, int reg);
diff --git a/target/riscv/machine.c b/target/riscv/machine.c
new file mode 100644
index 00..af2828a80a
--- /dev/null
+++ b/target/riscv/machine.c
@@ -0,0 +1,77 @@
+/*
+ * RISC-V VMState Description
+ *
+ * Copyright (c) 2020 Huawei Technologies Co., Ltd
+ *
+ * This program is free software; you can redistribute it and/or modify it
+ * under the terms and conditions of the GNU General Public License,
+ * version 2 or later, as published by the Free Software Foundation.
+ *
+ * This program is distributed in the hope it will be useful, but WITHOUT
+ * ANY WARRANTY; without even the implied warranty of MERCHANTABILITY or
+ * FITNESS FOR A PARTICULAR PURPOSE.  See the GNU General Public License for
+ * more details.
+ *
+ * You should have received a copy of the GNU General Public License along with
+ * this program.  If not, see .
+ */
+
+#include "qemu/osdep.h"
+#include "cpu.h"
+#include "qemu/error-report.h"
+#include "sysemu/kvm.h"
+#include "migration/cpu.h"
+
+const VMStateDescription vmstate_riscv_cpu = {
+.name = "cpu",
+.version_id = 1,
+.minimum_version_id = 1,
+.fields = (VMStateField[]) {
+VMSTATE_UINTTL_ARRAY(env.gpr, RISCVCPU, 32),
+VMSTATE_UINT64_ARRAY(env.fpr, RISCVCPU, 32),
+VMSTATE_UINTTL(env.pc, RISCVCPU),
+VMSTATE_UINTTL(env.load_res, RISCVCPU),
+VMSTATE_UINTTL(env.load_val, RISCVCPU),
+VMSTATE_UINTTL(env.frm, RISCVCPU),
+VMSTATE_UINTTL(env.badaddr, RISCVCPU),
+VMSTATE_UINTTL(env.guest_phys_fault_addr, RISCVCPU),
+VMSTATE_UINTTL(env.priv_ver, RISCVCPU),
+VMSTATE_UINTTL(env.vext_ver, RISCVCPU),
+VMSTATE_UINTTL(env.misa, RISCVCPU),
+VMSTATE_UINTTL(env.misa_mask, RISCVCPU),
+VMSTATE_UINT32(env.features, RISCVCPU),
+VMSTATE_UINTTL(env.priv, RISCVCPU),
+VMSTATE_UINTTL(env.virt, RISCVCPU),
+VMSTATE_UINTTL(env.resetvec, RISCVCPU),
+VMSTATE_UINTTL(env.mhartid, RISCVCPU),
+VMSTATE_UINTTL(env.mstatus, RISCVCPU),
+VMSTATE_UINTTL(env.mip, RISCVCPU),
+VMSTATE_UINT32(env.miclaim, RISCVCPU),
+VMSTATE_UINTTL(env.mie, RISCVCPU),
+VMSTATE_UINTTL(env.mideleg, RISCVCPU),
+VMSTATE_UINTTL(env.sptbr, RISCVCPU),
+VMSTATE_UINTTL(env.satp, RISCVCPU),
+VMSTATE_UINTTL(env.sbadaddr, RISCVCPU),
+VMSTATE_UINTTL(env.mbadaddr, RISCVCPU),
+VMSTATE_UINTTL(env.medeleg, RISCVCPU),
+VMSTATE_UINTTL(env.stvec, RISCVCPU),
+VMSTATE_UINTTL(env.sepc, RISCVCPU),
+VMSTATE_UINTTL(env.scause, RISCVCPU),
+VMSTATE_UINTTL(env.mtvec, RISCVCPU),
+VMSTATE_UINTTL(env.mepc, RISCVCPU),
+VMSTATE_UINTTL(env.mcause, RISCVCPU),
+VMSTATE_UINTTL(env.mtval, RISCVCPU),
+VMSTATE_UINTTL(env.scounteren, RISCVCPU),
+VMSTATE_UINTTL(env.mcounteren, RISCVCPU),
+VMSTATE_UINTTL(env.sscratch, RISCVCPU),
+VMSTATE_UINTTL(env.mscratch, RISCVCPU),
+VMSTATE_UINT64(env.mfromhost, RISCVCPU),
+VMSTATE_UINT64(env.mtohost, RISCVCPU),
+VMSTATE_UINT64(env.timecmp, RISCVCPU),
+
+#ifdef TARGET_RISCV32
+VMSTATE_UINTTL(env.mstatush, RISCVCPU),
+#endif
+VMSTATE_END_OF_LIST()
+}
+};
diff --git a/target/riscv/meson.build b/target/riscv/meson.build
index abd647fea1..14a5c62dac 100644
--- a/target/riscv/meson.build
+++ b/target/riscv/meson.build
@@ -27,7 +27,8 @@ riscv_ss.add(files(
 riscv_softmmu_ss = ss

[PULL 34/39] device-core: use atomic_set on .realized property

[Paolo Bonzini]

From: Maxim Levitsky 

Some code might race with placement of new devices on a bus.
We currently first place a (unrealized) device on the bus
and then realize it.

As a workaround, users that scan the child device list, can
check the realized property to see if it is safe to access such a device.
Use an atomic write here too to aid with this.

A separate discussion is what to do with devices that are unrealized:
It looks like for this case we only call the hotplug handler's unplug
callback and its up to it to unrealize the device.
An atomic operation doesn't cause harm for this code path though.

Signed-off-by: Maxim Levitsky 
Reviewed-by: Stefan Hajnoczi 
Message-Id: <20200913160259.32145-6-mlevi...@redhat.com>
Signed-off-by: Paolo Bonzini 
Message-Id: <20201006123904.610658-10-mlevi...@redhat.com>
Signed-off-by: Paolo Bonzini 
---
 hw/core/qdev.c | 19 ++-
 include/hw/qdev-core.h |  2 ++
 2 files changed, 20 insertions(+), 1 deletion(-)

diff --git a/hw/core/qdev.c b/hw/core/qdev.c
index 59e5e710b7..fc4daa36fa 100644
--- a/hw/core/qdev.c
+++ b/hw/core/qdev.c
@@ -946,7 +946,25 @@ static void device_set_realized(Object *obj, bool value, 
Error **errp)
 }
}
 
+   qatomic_store_release(&dev->realized, value);
+
 } else if (!value && dev->realized) {
+
+/*
+ * Change the value so that any concurrent users are aware
+ * that the device is going to be unrealized
+ *
+ * TODO: change .realized property to enum that states
+ * each phase of the device realization/unrealization
+ */
+
+qatomic_set(&dev->realized, value);
+/*
+ * Ensure that concurrent users see this update prior to
+ * any other changes done by unrealize.
+ */
+smp_wmb();
+
 QLIST_FOREACH(bus, &dev->child_bus, sibling) {
 qbus_unrealize(bus);
 }
@@ -961,7 +979,6 @@ static void device_set_realized(Object *obj, bool value, 
Error **errp)
 }
 
 assert(local_err == NULL);
-dev->realized = value;
 return;
 
 child_realize_fail:
diff --git a/include/hw/qdev-core.h b/include/hw/qdev-core.h
index 2c6307e3ed..868973319e 100644
--- a/include/hw/qdev-core.h
+++ b/include/hw/qdev-core.h
@@ -163,6 +163,8 @@ struct NamedClockList {
 /**
  * DeviceState:
  * @realized: Indicates whether the device has been fully constructed.
+ *When accessed outsize big qemu lock, must be accessed with
+ *atomic_load_acquire()
  * @reset: ResettableState for the device; handled by Resettable interface.
  *
  * This structure should not be accessed directly.  We declare it here
-- 
2.26.2

[PULL 21/39] meson.build: Re-enable KVM support for MIPS

[Paolo Bonzini]

From: Huacai Chen 

After converting from configure to meson, KVM support is lost for MIPS,
so re-enable it in meson.build.

Fixes: fdb75aeff7c212e1afaaa3a43 ("configure: remove target configuration")
Fixes: 8a19980e3fc42239aae054bc9 ("configure: move accelerator logic to meson")
Cc: aolo Bonzini 
Signed-off-by: Huacai Chen 
Message-Id: <1602059975-10115-3-git-send-email-che...@lemote.com>
Signed-off-by: Paolo Bonzini 
---
 meson.build | 2 ++
 1 file changed, 2 insertions(+)

diff --git a/meson.build b/meson.build
index 26230614ba..0c676af194 100644
--- a/meson.build
+++ b/meson.build
@@ -59,6 +59,8 @@ elif cpu == 's390x'
   kvm_targets = ['s390x-softmmu']
 elif cpu in ['ppc', 'ppc64']
   kvm_targets = ['ppc-softmmu', 'ppc64-softmmu']
+elif cpu in ['mips', 'mips64']
+  kvm_targets = ['mips-softmmu', 'mipsel-softmmu', 'mips64-softmmu', 
'mips64el-softmmu']
 else
   kvm_targets = []
 endif
-- 
2.26.2

[PULL 25/39] device-plug-test: use qtest_qmp to send the device_del command

[Paolo Bonzini]

Simplify the code now that events are buffered.  There is no need
anymore to separate sending the command and retrieving the response.

Signed-off-by: Paolo Bonzini 
---
 tests/qtest/device-plug-test.c | 32 +---
 1 file changed, 9 insertions(+), 23 deletions(-)

diff --git a/tests/qtest/device-plug-test.c b/tests/qtest/device-plug-test.c
index a2247856be..559d47727a 100644
--- a/tests/qtest/device-plug-test.c
+++ b/tests/qtest/device-plug-test.c
@@ -15,26 +15,17 @@
 #include "qapi/qmp/qdict.h"
 #include "qapi/qmp/qstring.h"
 
-static void device_del_start(QTestState *qtest, const char *id)
+static void device_del(QTestState *qtest, const char *id)
 {
-qtest_qmp_send(qtest,
-   "{'execute': 'device_del', 'arguments': { 'id': %s } }", 
id);
-}
+QDict *resp;
 
-static void device_del_finish(QTestState *qtest)
-{
-QDict *resp = qtest_qmp_receive_dict(qtest);
+resp = qtest_qmp(qtest,
+ "{'execute': 'device_del', 'arguments': { 'id': %s } }", 
id);
 
 g_assert(qdict_haskey(resp, "return"));
 qobject_unref(resp);
 }
 
-static void device_del_request(QTestState *qtest, const char *id)
-{
-device_del_start(qtest, id);
-device_del_finish(qtest);
-}
-
 static void system_reset(QTestState *qtest)
 {
 QDict *resp;
@@ -79,7 +70,7 @@ static void test_pci_unplug_request(void)
  * be processed. However during system reset, the removal will be
  * handled, removing the device.
  */
-device_del_request(qtest, "dev0");
+device_del(qtest, "dev0");
 system_reset(qtest);
 wait_device_deleted_event(qtest, "dev0");
 
@@ -90,13 +81,8 @@ static void test_ccw_unplug(void)
 {
 QTestState *qtest = qtest_initf("-device virtio-balloon-ccw,id=dev0");
 
-/*
- * The DEVICE_DELETED events will be sent before the command
- * completes.
- */
-device_del_start(qtest, "dev0");
+device_del(qtest, "dev0");
 wait_device_deleted_event(qtest, "dev0");
-device_del_finish(qtest);
 
 qtest_quit(qtest);
 }
@@ -109,7 +95,7 @@ static void test_spapr_cpu_unplug_request(void)
 "-device 
power9_v2.0-spapr-cpu-core,core-id=1,id=dev0");
 
 /* similar to test_pci_unplug_request */
-device_del_request(qtest, "dev0");
+device_del(qtest, "dev0");
 system_reset(qtest);
 wait_device_deleted_event(qtest, "dev0");
 
@@ -125,7 +111,7 @@ static void test_spapr_memory_unplug_request(void)
 "-device pc-dimm,id=dev0,memdev=mem0");
 
 /* similar to test_pci_unplug_request */
-device_del_request(qtest, "dev0");
+device_del(qtest, "dev0");
 system_reset(qtest);
 wait_device_deleted_event(qtest, "dev0");
 
@@ -139,7 +125,7 @@ static void test_spapr_phb_unplug_request(void)
 qtest = qtest_initf("-device spapr-pci-host-bridge,index=1,id=dev0");
 
 /* similar to test_pci_unplug_request */
-device_del_request(qtest, "dev0");
+device_del(qtest, "dev0");
 system_reset(qtest);
 wait_device_deleted_event(qtest, "dev0");
 
-- 
2.26.2

[PULL 36/39] scsi/scsi_bus: Add scsi_device_get

[Paolo Bonzini]

From: Maxim Levitsky 

Add scsi_device_get which finds the scsi device
and takes a reference to it.

Suggested-by: Stefan Hajnoczi 
Signed-off-by: Maxim Levitsky 
Message-Id: <20200913160259.32145-8-mlevi...@redhat.com>
Signed-off-by: Paolo Bonzini 
Message-Id: <20201006123904.610658-12-mlevi...@redhat.com>
Signed-off-by: Paolo Bonzini 
---
 hw/scsi/scsi-bus.c | 11 +++
 include/hw/scsi/scsi.h |  1 +
 2 files changed, 12 insertions(+)

diff --git a/hw/scsi/scsi-bus.c b/hw/scsi/scsi-bus.c
index 7599113efe..eda8cb7e70 100644
--- a/hw/scsi/scsi-bus.c
+++ b/hw/scsi/scsi-bus.c
@@ -73,6 +73,17 @@ SCSIDevice *scsi_device_find(SCSIBus *bus, int channel, int 
id, int lun)
 return do_scsi_device_find(bus, channel, id, lun, false);
 }
 
+SCSIDevice *scsi_device_get(SCSIBus *bus, int channel, int id, int lun)
+{
+SCSIDevice *d;
+RCU_READ_LOCK_GUARD();
+d = do_scsi_device_find(bus, channel, id, lun, false);
+if (d) {
+object_ref(d);
+}
+return d;
+}
+
 static void scsi_device_realize(SCSIDevice *s, Error **errp)
 {
 SCSIDeviceClass *sc = SCSI_DEVICE_GET_CLASS(s);
diff --git a/include/hw/scsi/scsi.h b/include/hw/scsi/scsi.h
index 7a55cdbd74..09fa5c9d2a 100644
--- a/include/hw/scsi/scsi.h
+++ b/include/hw/scsi/scsi.h
@@ -190,6 +190,7 @@ int scsi_device_get_sense(SCSIDevice *dev, uint8_t *buf, 
int len, bool fixed);
 int scsi_SG_IO_FROM_DEV(BlockBackend *blk, uint8_t *cmd, uint8_t cmd_size,
 uint8_t *buf, uint8_t buf_size);
 SCSIDevice *scsi_device_find(SCSIBus *bus, int channel, int target, int lun);
+SCSIDevice *scsi_device_get(SCSIBus *bus, int channel, int target, int lun);
 
 /* scsi-generic.c. */
 extern const SCSIReqOps scsi_generic_req_ops;
-- 
2.26.2

[PULL 28/39] qemu-iotests, qtest: rewrite test 067 as a qtest

[Paolo Bonzini]

Test 067 from qemu-iotests is executing QMP commands to hotplug
and hot-unplug disks, devices and blockdevs.  Because the power
of the text-based test harness is limited, it is actually limiting
the checks that it does, for example by skipping DEVICE_DELETED
events.

tests/qtest already has a similar test, drive_del-test.c.
We can merge them, and even reuse some of the existing code in
drive_del-test.c.  This will improve the quality of the test by
covering DEVICE_DELETED events and testing multiple architectures
(therefore covering multiple PCI hotplug mechanisms as well as s390x
virtio-ccw).

The only difference is that the new test will always use null-co:// for
the medium rather than qcow2 or raw, but this should be irrelevant for
what the test is covering.  For example there are no "qemu-img check"
runs in 067 that would check that the file is properly closed.

The new tests requires PCI hot-plug support, so drive_del-test
is moved from qemu-system-ppc to qemu-system-ppc64.

Reviewed-by: Kevin Wolf 
Signed-off-by: Paolo Bonzini 
---
 .gitlab-ci.yml   |   2 +-
 tests/qemu-iotests/067   | 157 -
 tests/qemu-iotests/067.out   | 414 ---
 tests/qemu-iotests/group |   2 +-
 tests/qtest/drive_del-test.c | 211 --
 tests/qtest/meson.build  |   4 +-
 6 files changed, 191 insertions(+), 599 deletions(-)
 delete mode 100755 tests/qemu-iotests/067
 delete mode 100644 tests/qemu-iotests/067.out

diff --git a/.gitlab-ci.yml b/.gitlab-ci.yml
index a51c89554f..a4cf87481e 100644
--- a/.gitlab-ci.yml
+++ b/.gitlab-ci.yml
@@ -232,7 +232,7 @@ build-tcg-disabled:
 - ./check -raw 001 002 003 004 005 008 009 010 011 012 021 025 032 033 048
 052 063 077 086 101 104 106 113 148 150 151 152 157 159 160 163
 170 171 183 184 192 194 197 208 215 221 222 226 227 236 253 277
-- ./check -qcow2 028 051 056 057 058 065 067 068 082 085 091 095 096 102 
122
+- ./check -qcow2 028 051 056 057 058 065 068 082 085 091 095 096 102 122
 124 132 139 142 144 145 151 152 155 157 165 194 196 197 200 202
 208 209 215 216 218 222 227 234 246 247 248 250 254 255 257 258
 260 261 262 263 264 270 272 273 277 279
diff --git a/tests/qemu-iotests/067 b/tests/qemu-iotests/067
deleted file mode 100755
index a63be9cabf..00
--- a/tests/qemu-iotests/067
+++ /dev/null
@@ -1,157 +0,0 @@
-#!/usr/bin/env bash
-#
-# Test automatic deletion of BDSes created by -drive/drive_add
-#
-# Copyright (C) 2013 Red Hat, Inc.
-#
-# This program is free software; you can redistribute it and/or modify
-# it under the terms of the GNU General Public License as published by
-# the Free Software Foundation; either version 2 of the License, or
-# (at your option) any later version.
-#
-# This program is distributed in the hope that it will be useful,
-# but WITHOUT ANY WARRANTY; without even the implied warranty of
-# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
-# GNU General Public License for more details.
-#
-# You should have received a copy of the GNU General Public License
-# along with this program.  If not, see .
-#
-
-# creator
-owner=kw...@redhat.com
-
-seq=`basename $0`
-echo "QA output created by $seq"
-
-status=1   # failure is the default!
-
-# get standard environment, filters and checks
-. ./common.rc
-. ./common.filter
-
-_supported_fmt qcow2
-_supported_proto file
-# Because anything other than 16 would change the output of query-block,
-# and external data files would change the output of
-# query-named-block-nodes
-_unsupported_imgopts 'refcount_bits=\([^1]\|.\([^6]\|$\)\)' data_file
-
-do_run_qemu()
-{
-echo Testing: "$@"
-$QEMU -nographic -qmp-pretty stdio -serial none "$@"
-echo
-}
-
-# Remove QMP events from (pretty-printed) output. Doesn't handle
-# nested dicts correctly, but we don't get any of those in this test.
-_filter_qmp_events()
-{
-tr '\n' '\t' | sed -e \
-   
's/{\s*"timestamp":\s*{[^}]*},\s*"event":[^,}]*\(,\s*"data":\s*{[^}]*}\)\?\s*}\s*//g'
 \
-   | tr '\t' '\n'
-}
-
-run_qemu()
-{
-do_run_qemu "$@" 2>&1 | _filter_testdir | _filter_qmp | _filter_qemu \
-  | _filter_actual_image_size \
-  | _filter_generated_node_ids | _filter_qmp_events \
-  | _filter_img_info
-}
-
-size=128M
-
-_make_test_img $size
-
-echo
-echo === -drive/-device and device_del ===
-echo
-
-run_qemu -drive file=$TEST_IMG,format=$IMGFMT,if=none,id=disk -device 
virtio-blk,drive=disk,id=virtio0 

[PATCH V2 5/5] target/riscv: Add sifive_plic vmstate

[Yifei Jiang]

Add sifive_plic vmstate for supporting sifive_plic migration.
Current vmstate framework only supports one structure parameter
as num field to describe variable length arrays, so introduce
num_enables.

Signed-off-by: Yifei Jiang 
Signed-off-by: Yipeng Yin 
---
 hw/intc/sifive_plic.c | 26 +-
 hw/intc/sifive_plic.h |  1 +
 2 files changed, 26 insertions(+), 1 deletion(-)

diff --git a/hw/intc/sifive_plic.c b/hw/intc/sifive_plic.c
index f42fd695d8..97a1a27a9a 100644
--- a/hw/intc/sifive_plic.c
+++ b/hw/intc/sifive_plic.c
@@ -30,6 +30,7 @@
 #include "hw/intc/sifive_plic.h"
 #include "target/riscv/cpu.h"
 #include "sysemu/sysemu.h"
+#include "migration/vmstate.h"
 
 #define RISCV_DEBUG_PLIC 0
 
@@ -448,11 +449,12 @@ static void sifive_plic_realize(DeviceState *dev, Error 
**errp)
   TYPE_SIFIVE_PLIC, plic->aperture_size);
 parse_hart_config(plic);
 plic->bitfield_words = (plic->num_sources + 31) >> 5;
+plic->num_enables = plic->bitfield_words * plic->num_addrs;
 plic->source_priority = g_new0(uint32_t, plic->num_sources);
 plic->target_priority = g_new(uint32_t, plic->num_addrs);
 plic->pending = g_new0(uint32_t, plic->bitfield_words);
 plic->claimed = g_new0(uint32_t, plic->bitfield_words);
-plic->enable = g_new0(uint32_t, plic->bitfield_words * plic->num_addrs);
+plic->enable = g_new0(uint32_t, plic->num_enables);
 sysbus_init_mmio(SYS_BUS_DEVICE(dev), &plic->mmio);
 qdev_init_gpio_in(dev, sifive_plic_irq_request, plic->num_sources);
 
@@ -472,12 +474,34 @@ static void sifive_plic_realize(DeviceState *dev, Error 
**errp)
 msi_nonbroken = true;
 }
 
+static const VMStateDescription vmstate_sifive_plic = {
+.name = "riscv_sifive_plic",
+.version_id = 1,
+.minimum_version_id = 1,
+.fields = (VMStateField[]) {
+VMSTATE_VARRAY_UINT32(source_priority, SiFivePLICState,
+  num_sources, 0,
+  vmstate_info_uint32, uint32_t),
+VMSTATE_VARRAY_UINT32(target_priority, SiFivePLICState,
+  num_addrs, 0,
+  vmstate_info_uint32, uint32_t),
+VMSTATE_VARRAY_UINT32(pending, SiFivePLICState, bitfield_words, 0,
+  vmstate_info_uint32, uint32_t),
+VMSTATE_VARRAY_UINT32(claimed, SiFivePLICState, bitfield_words, 0,
+  vmstate_info_uint32, uint32_t),
+VMSTATE_VARRAY_UINT32(enable, SiFivePLICState, num_enables, 0,
+  vmstate_info_uint32, uint32_t),
+VMSTATE_END_OF_LIST()
+}
+};
+
 static void sifive_plic_class_init(ObjectClass *klass, void *data)
 {
 DeviceClass *dc = DEVICE_CLASS(klass);
 
 device_class_set_props(dc, sifive_plic_properties);
 dc->realize = sifive_plic_realize;
+dc->vmsd = &vmstate_sifive_plic;
 }
 
 static const TypeInfo sifive_plic_info = {
diff --git a/hw/intc/sifive_plic.h b/hw/intc/sifive_plic.h
index b75b1f145d..1e451a270c 100644
--- a/hw/intc/sifive_plic.h
+++ b/hw/intc/sifive_plic.h
@@ -52,6 +52,7 @@ struct SiFivePLICState {
 uint32_t num_addrs;
 uint32_t num_harts;
 uint32_t bitfield_words;
+uint32_t num_enables;
 PLICAddr *addr_config;
 uint32_t *source_priority;
 uint32_t *target_priority;
-- 
2.19.1

[PULL 32/39] device-core: use RCU for list of children of a bus

[Paolo Bonzini]

From: Maxim Levitsky 

This fixes the race between device emulation code that tries to find
a child device to dispatch the request to (e.g a scsi disk),
and hotplug of a new device to that bus.

Note that this doesn't convert all the readers of the list
but only these that might go over that list without BQL held.

This is a very small first step to make this code thread safe.

Suggested-by: Paolo Bonzini 
Signed-off-by: Maxim Levitsky 
Reviewed-by: Stefan Hajnoczi 
Message-Id: <20200913160259.32145-5-mlevi...@redhat.com>
[Use RCU_READ_LOCK_GUARD in more places, adjust testcase now that
 the delay in DEVICE_DELETED due to RCU is more consistent. - Paolo]
Signed-off-by: Paolo Bonzini 
Message-Id: <20201006123904.610658-9-mlevi...@redhat.com>
Signed-off-by: Paolo Bonzini 
---
 hw/core/bus.c  | 28 +---
 hw/core/qdev.c | 37 +++--
 hw/scsi/scsi-bus.c | 12 +---
 hw/scsi/virtio-scsi.c  |  6 +-
 include/hw/qdev-core.h |  9 +
 5 files changed, 63 insertions(+), 29 deletions(-)

diff --git a/hw/core/bus.c b/hw/core/bus.c
index 6b987b6946..a0483859ae 100644
--- a/hw/core/bus.c
+++ b/hw/core/bus.c
@@ -49,12 +49,14 @@ int qbus_walk_children(BusState *bus,
 }
 }
 
-QTAILQ_FOREACH(kid, &bus->children, sibling) {
-err = qdev_walk_children(kid->child,
- pre_devfn, pre_busfn,
- post_devfn, post_busfn, opaque);
-if (err < 0) {
-return err;
+WITH_RCU_READ_LOCK_GUARD() {
+QTAILQ_FOREACH_RCU(kid, &bus->children, sibling) {
+err = qdev_walk_children(kid->child,
+ pre_devfn, pre_busfn,
+ post_devfn, post_busfn, opaque);
+if (err < 0) {
+return err;
+}
 }
 }
 
@@ -90,8 +92,10 @@ static void bus_reset_child_foreach(Object *obj, 
ResettableChildCallback cb,
 BusState *bus = BUS(obj);
 BusChild *kid;
 
-QTAILQ_FOREACH(kid, &bus->children, sibling) {
-cb(OBJECT(kid->child), opaque, type);
+WITH_RCU_READ_LOCK_GUARD() {
+QTAILQ_FOREACH_RCU(kid, &bus->children, sibling) {
+cb(OBJECT(kid->child), opaque, type);
+}
 }
 }
 
@@ -194,9 +198,11 @@ static void bus_set_realized(Object *obj, bool value, 
Error **errp)
 
 /* TODO: recursive realization */
 } else if (!value && bus->realized) {
-QTAILQ_FOREACH(kid, &bus->children, sibling) {
-DeviceState *dev = kid->child;
-qdev_unrealize(dev);
+WITH_RCU_READ_LOCK_GUARD() {
+QTAILQ_FOREACH_RCU(kid, &bus->children, sibling) {
+DeviceState *dev = kid->child;
+qdev_unrealize(dev);
+}
 }
 if (bc->unrealize) {
 bc->unrealize(bus);
diff --git a/hw/core/qdev.c b/hw/core/qdev.c
index 74db78df36..59e5e710b7 100644
--- a/hw/core/qdev.c
+++ b/hw/core/qdev.c
@@ -51,6 +51,12 @@ const VMStateDescription *qdev_get_vmsd(DeviceState *dev)
 return dc->vmsd;
 }
 
+static void bus_free_bus_child(BusChild *kid)
+{
+object_unref(OBJECT(kid->child));
+g_free(kid);
+}
+
 static void bus_remove_child(BusState *bus, DeviceState *child)
 {
 BusChild *kid;
@@ -60,15 +66,16 @@ static void bus_remove_child(BusState *bus, DeviceState 
*child)
 char name[32];
 
 snprintf(name, sizeof(name), "child[%d]", kid->index);
-QTAILQ_REMOVE(&bus->children, kid, sibling);
+QTAILQ_REMOVE_RCU(&bus->children, kid, sibling);
 
 bus->num_children--;
 
 /* This gives back ownership of kid->child back to us.  */
 object_property_del(OBJECT(bus), name);
-object_unref(OBJECT(kid->child));
-g_free(kid);
-return;
+
+/* free the bus kid, when it is safe to do so*/
+call_rcu(kid, bus_free_bus_child, rcu);
+break;
 }
 }
 }
@@ -83,7 +90,7 @@ static void bus_add_child(BusState *bus, DeviceState *child)
 kid->child = child;
 object_ref(OBJECT(kid->child));
 
-QTAILQ_INSERT_HEAD(&bus->children, kid, sibling);
+QTAILQ_INSERT_HEAD_RCU(&bus->children, kid, sibling);
 
 /* This transfers ownership of kid->child to the property.  */
 snprintf(name, sizeof(name), "child[%d]", kid->index);
@@ -672,17 +679,19 @@ DeviceState *qdev_find_recursive(BusState *bus, const 
char *id)
 DeviceState *ret;
 BusState *child;
 
-QTAILQ_FOREACH(kid, &bus->children, sibling) {
-DeviceState *dev = kid->child;
+WITH_RCU_READ_LOCK_GUARD() {
+QTAILQ_FOREACH_RCU(kid, &bus->children, sibling) {
+DeviceState *dev = kid->child;
 
-if (dev->id && strcmp(dev->id, id) == 0) {
-return dev;
-}
+if (dev->id && strcmp(dev->id, id) == 0) {
+return dev;
+

[PULL 27/39] qtest: check that drives are really appearing and disappearing

[Paolo Bonzini]

Do not just trust the HMP commands to create and delete the drive, use
query-block to check that this is actually the case.

Reviewed-by: Kevin Wolf 
Signed-off-by: Paolo Bonzini 
---
 tests/qtest/drive_del-test.c | 32 +++-
 1 file changed, 31 insertions(+), 1 deletion(-)

diff --git a/tests/qtest/drive_del-test.c b/tests/qtest/drive_del-test.c
index 9d20a1ed8b..ff772b3671 100644
--- a/tests/qtest/drive_del-test.c
+++ b/tests/qtest/drive_del-test.c
@@ -14,20 +14,49 @@
 #include "libqos/libqtest.h"
 #include "libqos/virtio.h"
 #include "qapi/qmp/qdict.h"
+#include "qapi/qmp/qlist.h"
+
+static bool has_drive(QTestState *qts)
+{
+QDict *response;
+QList *ret;
+QListEntry *entry;
+bool found;
+
+response = qtest_qmp(qts, "{'execute': 'query-block'}");
+g_assert(response && qdict_haskey(response, "return"));
+ret = qdict_get_qlist(response, "return");
+
+found = false;
+QLIST_FOREACH_ENTRY(ret, entry) {
+QDict *entry_dict = qobject_to(QDict, entry->value);
+if (!strcmp(qdict_get_str(entry_dict, "device"), "drive0")) {
+found = true;
+break;
+}
+}
+
+qobject_unref(response);
+return found;
+}
 
 static void drive_add(QTestState *qts)
 {
 char *resp = qtest_hmp(qts, "drive_add 0 if=none,id=drive0");
 
 g_assert_cmpstr(resp, ==, "OK\r\n");
+g_assert(has_drive(qts));
 g_free(resp);
 }
 
 static void drive_del(QTestState *qts)
 {
-char *resp = qtest_hmp(qts, "drive_del drive0");
+char *resp;
 
+g_assert(has_drive(qts));
+resp = qtest_hmp(qts, "drive_del drive0");
 g_assert_cmpstr(resp, ==, "");
+g_assert(!has_drive(qts));
 g_free(resp);
 }
 
@@ -130,6 +159,7 @@ static void test_drive_del_device_del(void)
  */
 drive_del(qts);
 device_del(qts);
+g_assert(!has_drive(qts));
 
 qtest_quit(qts);
 }
-- 
2.26.2

[PULL 38/39] scsi/scsi_bus: fix races in REPORT LUNS

[Paolo Bonzini]

From: Maxim Levitsky 

Currently scsi_target_emulate_report_luns iterates over the child device list
twice, and there is no guarantee that this list is the same in both iterations.

The reason for iterating twice is that the first iteration calculates
how much memory to allocate.  However if we use a dynamic array we can
avoid iterating twice, and therefore we avoid this race.

Buglink: https://bugzilla.redhat.com/show_bug.cgi?id=1866707

Signed-off-by: Maxim Levitsky 
Reviewed-by: Stefan Hajnoczi 
Message-Id: <20200913160259.32145-10-mlevi...@redhat.com>
Signed-off-by: Paolo Bonzini 
Message-Id: <20201006123904.610658-14-mlevi...@redhat.com>
Signed-off-by: Paolo Bonzini 
---
 hw/scsi/scsi-bus.c | 68 ++
 1 file changed, 33 insertions(+), 35 deletions(-)

diff --git a/hw/scsi/scsi-bus.c b/hw/scsi/scsi-bus.c
index eda8cb7e70..b901e701f0 100644
--- a/hw/scsi/scsi-bus.c
+++ b/hw/scsi/scsi-bus.c
@@ -438,19 +438,23 @@ struct SCSITargetReq {
 static void store_lun(uint8_t *outbuf, int lun)
 {
 if (lun < 256) {
+/* Simple logical unit addressing method*/
+outbuf[0] = 0;
 outbuf[1] = lun;
-return;
+} else {
+/* Flat space addressing method */
+outbuf[0] = 0x40 | (lun >> 8);
+outbuf[1] = (lun & 255);
 }
-outbuf[1] = (lun & 255);
-outbuf[0] = (lun >> 8) | 0x40;
 }
 
 static bool scsi_target_emulate_report_luns(SCSITargetReq *r)
 {
 BusChild *kid;
-int i, len, n;
 int channel, id;
-bool found_lun0;
+uint8_t tmp[8] = {0};
+int len = 0;
+GByteArray *buf;
 
 if (r->req.cmd.xfer < 16) {
 return false;
@@ -458,46 +462,40 @@ static bool scsi_target_emulate_report_luns(SCSITargetReq 
*r)
 if (r->req.cmd.buf[2] > 2) {
 return false;
 }
+
+/* reserve space for 63 LUNs*/
+buf = g_byte_array_sized_new(512);
+
 channel = r->req.dev->channel;
 id = r->req.dev->id;
-found_lun0 = false;
-n = 0;
 
-RCU_READ_LOCK_GUARD();
+/* add size (will be updated later to correct value */
+g_byte_array_append(buf, tmp, 8);
+len += 8;
 
-QTAILQ_FOREACH_RCU(kid, &r->req.bus->qbus.children, sibling) {
-DeviceState *qdev = kid->child;
-SCSIDevice *dev = SCSI_DEVICE(qdev);
+/* add LUN0 */
+g_byte_array_append(buf, tmp, 8);
+len += 8;
 
-if (dev->channel == channel && dev->id == id) {
-if (dev->lun == 0) {
-found_lun0 = true;
+WITH_RCU_READ_LOCK_GUARD() {
+QTAILQ_FOREACH_RCU(kid, &r->req.bus->qbus.children, sibling) {
+DeviceState *qdev = kid->child;
+SCSIDevice *dev = SCSI_DEVICE(qdev);
+
+if (dev->channel == channel && dev->id == id && dev->lun != 0) {
+store_lun(tmp, dev->lun);
+g_byte_array_append(buf, tmp, 8);
+len += 8;
 }
-n += 8;
 }
 }
-if (!found_lun0) {
-n += 8;
-}
-
-scsi_target_alloc_buf(&r->req, n + 8);
-
-len = MIN(n + 8, r->req.cmd.xfer & ~7);
-memset(r->buf, 0, len);
-stl_be_p(&r->buf[0], n);
-i = found_lun0 ? 8 : 16;
-QTAILQ_FOREACH_RCU(kid, &r->req.bus->qbus.children, sibling) {
-DeviceState *qdev = kid->child;
-SCSIDevice *dev = SCSI_DEVICE(qdev);
 
-if (dev->channel == channel && dev->id == id) {
-store_lun(&r->buf[i], dev->lun);
-i += 8;
-}
-}
+r->buf_len = len;
+r->buf = g_byte_array_free(buf, FALSE);
+r->len = MIN(len, r->req.cmd.xfer & ~7);
 
-assert(i == n + 8);
-r->len = len;
+/* store the LUN list length */
+stl_be_p(&r->buf[0], len - 8);
 return true;
 }
 
-- 
2.26.2

[PULL 35/39] scsi/scsi-bus: scsi_device_find: don't return unrealized devices

[Paolo Bonzini]

The device core first places a device on the bus and then realizes it.
Make scsi_device_find avoid returing such devices to avoid
races in drivers that use an iothread (currently virtio-scsi)

Bugzilla: https://bugzilla.redhat.com/show_bug.cgi?id=1812399

Suggested-by: Paolo Bonzini 
Signed-off-by: Maxim Levitsky 
Reviewed-by: Stefan Hajnoczi 
Message-Id: <20200913160259.32145-7-mlevi...@redhat.com>
Signed-off-by: Paolo Bonzini 
Message-Id: <20201006123904.610658-11-mlevi...@redhat.com>
Signed-off-by: Paolo Bonzini 
---
 hw/scsi/scsi-bus.c | 83 +-
 1 file changed, 53 insertions(+), 30 deletions(-)

diff --git a/hw/scsi/scsi-bus.c b/hw/scsi/scsi-bus.c
index 4ab9811cd8..7599113efe 100644
--- a/hw/scsi/scsi-bus.c
+++ b/hw/scsi/scsi-bus.c
@@ -24,6 +24,55 @@ static void scsi_target_free_buf(SCSIRequest *req);
 
 static int next_scsi_bus;
 
+static SCSIDevice *do_scsi_device_find(SCSIBus *bus,
+   int channel, int id, int lun,
+   bool include_unrealized)
+{
+BusChild *kid;
+SCSIDevice *retval = NULL;
+
+QTAILQ_FOREACH_RCU(kid, &bus->qbus.children, sibling) {
+DeviceState *qdev = kid->child;
+SCSIDevice *dev = SCSI_DEVICE(qdev);
+
+if (dev->channel == channel && dev->id == id) {
+if (dev->lun == lun) {
+retval = dev;
+break;
+}
+
+/*
+ * If we don't find exact match (channel/bus/lun),
+ * we will return the first device which matches channel/bus
+ */
+
+if (!retval) {
+retval = dev;
+}
+}
+}
+
+/*
+ * This function might run on the IO thread and we might race against
+ * main thread hot-plugging the device.
+ * We assume that as soon as .realized is set to true we can let
+ * the user access the device.
+ */
+
+if (retval && !include_unrealized &&
+!qatomic_load_acquire(&retval->qdev.realized)) {
+retval = NULL;
+}
+
+return retval;
+}
+
+SCSIDevice *scsi_device_find(SCSIBus *bus, int channel, int id, int lun)
+{
+RCU_READ_LOCK_GUARD();
+return do_scsi_device_find(bus, channel, id, lun, false);
+}
+
 static void scsi_device_realize(SCSIDevice *s, Error **errp)
 {
 SCSIDeviceClass *sc = SCSI_DEVICE_GET_CLASS(s);
@@ -137,7 +186,10 @@ static bool scsi_bus_is_address_free(SCSIBus *bus,
 int channel, int target, int lun,
 SCSIDevice **p_dev)
 {
-SCSIDevice *d = scsi_device_find(bus, channel, target, lun);
+SCSIDevice *d;
+
+RCU_READ_LOCK_GUARD();
+d = do_scsi_device_find(bus, channel, target, lun, true);
 if (d && d->lun == lun) {
 if (p_dev) {
 *p_dev = d;
@@ -1570,35 +1622,6 @@ static char *scsibus_get_fw_dev_path(DeviceState *dev)
qdev_fw_name(dev), d->id, d->lun);
 }
 
-SCSIDevice *scsi_device_find(SCSIBus *bus, int channel, int id, int lun)
-{
-BusChild *kid;
-SCSIDevice *target_dev = NULL;
-
-RCU_READ_LOCK_GUARD();
-QTAILQ_FOREACH_RCU(kid, &bus->qbus.children, sibling) {
-DeviceState *qdev = kid->child;
-SCSIDevice *dev = SCSI_DEVICE(qdev);
-
-if (dev->channel == channel && dev->id == id) {
-if (dev->lun == lun) {
-return dev;
-}
-
-/*
- * If we don't find exact match (channel/bus/lun),
- * we will return the first device which matches channel/bus
- */
-
-if (!target_dev) {
-target_dev = dev;
-}
-}
-}
-
-return target_dev;
-}
-
 /* SCSI request list.  For simplicity, pv points to the whole device */
 
 static int put_scsi_requests(QEMUFile *f, void *pv, size_t size,
-- 
2.26.2

[PULL 24/39] qtest: remove qtest_qmp_receive_success

[Paolo Bonzini]

From: Maxim Levitsky 

The purpose of qtest_qmp_receive_success was mostly to process events
that arrived between the issueing of a command and the "return"
line from QMP.  This is now handled by the buffering of events
that libqtest performs automatically.

Signed-off-by: Paolo Bonzini 
Signed-off-by: Maxim Levitsky 
---
 tests/qtest/libqos/libqtest.h   | 17 ---
 tests/qtest/libqtest.c  | 53 -
 tests/qtest/migration-helpers.c | 25 
 3 files changed, 25 insertions(+), 70 deletions(-)

diff --git a/tests/qtest/libqos/libqtest.h b/tests/qtest/libqos/libqtest.h
index a2e3961792..64bb1cd9eb 100644
--- a/tests/qtest/libqos/libqtest.h
+++ b/tests/qtest/libqos/libqtest.h
@@ -240,23 +240,6 @@ QDict *qtest_qmp_eventwait_ref(QTestState *s, const char 
*event);
  */
 QDict *qtest_qmp_event_ref(QTestState *s, const char *event);
 
-/**
- * qtest_qmp_receive_success:
- * @s: #QTestState instance to operate on
- * @event_cb: Event callback
- * @opaque: Argument for @event_cb
- *
- * Poll QMP messages until a command success response is received.
- * If @event_cb, call it for each event received, passing @opaque,
- * the event's name and data.
- * Return the success response's "return" member.
- */
-QDict *qtest_qmp_receive_success(QTestState *s,
- void (*event_cb)(void *opaque,
-  const char *name,
-  QDict *data),
- void *opaque);
-
 /**
  * qtest_hmp:
  * @s: #QTestState instance to operate on.
diff --git a/tests/qtest/libqtest.c b/tests/qtest/libqtest.c
index d4c49a52ff..baac667b8d 100644
--- a/tests/qtest/libqtest.c
+++ b/tests/qtest/libqtest.c
@@ -1291,35 +1291,6 @@ void qtest_cb_for_every_machine(void (*cb)(const char 
*machine),
 qobject_unref(response);
 }
 
-QDict *qtest_qmp_receive_success(QTestState *s,
- void (*event_cb)(void *opaque,
-  const char *event,
-  QDict *data),
- void *opaque)
-{
-QDict *response, *ret, *data;
-const char *event;
-
-for (;;) {
-response = qtest_qmp_receive_dict(s);
-g_assert(!qdict_haskey(response, "error"));
-ret = qdict_get_qdict(response, "return");
-if (ret) {
-break;
-}
-event = qdict_get_str(response, "event");
-data = qdict_get_qdict(response, "data");
-if (event_cb) {
-event_cb(opaque, event, data);
-}
-qobject_unref(response);
-}
-
-qobject_ref(ret);
-qobject_unref(response);
-return ret;
-}
-
 /*
  * Generic hot-plugging test via the device_add QMP commands.
  */
@@ -1355,13 +1326,6 @@ void qtest_qmp_device_add(QTestState *qts, const char 
*driver, const char *id,
 qobject_unref(args);
 }
 
-static void device_deleted_cb(void *opaque, const char *name, QDict *data)
-{
-bool *got_event = opaque;
-
-g_assert_cmpstr(name, ==, "DEVICE_DELETED");
-*got_event = true;
-}
 
 /*
  * Generic hot-unplugging test via the device_del QMP command.
@@ -1378,24 +1342,17 @@ static void device_deleted_cb(void *opaque, const char 
*name, QDict *data)
  * and this one:
  *
  * {"return": {}}
- *
- * But the order of arrival may vary - so we've got to detect both.
  */
 void qtest_qmp_device_del(QTestState *qts, const char *id)
 {
-bool got_event = false;
 QDict *rsp;
 
-qtest_qmp_send(qts, "{'execute': 'device_del', 'arguments': {'id': %s}}",
-   id);
-rsp = qtest_qmp_receive_success(qts, device_deleted_cb, &got_event);
+rsp = qtest_qmp(qts, "{'execute': 'device_del', 'arguments': {'id': %s}}",
+id);
+
+g_assert(qdict_haskey(rsp, "return"));
 qobject_unref(rsp);
-if (!got_event) {
-rsp = qtest_qmp_receive_dict(qts);
-g_assert_cmpstr(qdict_get_try_str(rsp, "event"),
-==, "DEVICE_DELETED");
-qobject_unref(rsp);
-}
+qtest_qmp_eventwait(qts, "DEVICE_DELETED");
 }
 
 bool qmp_rsp_is_err(QDict *rsp)
diff --git a/tests/qtest/migration-helpers.c b/tests/qtest/migration-helpers.c
index 516093b39a..b799dbafb7 100644
--- a/tests/qtest/migration-helpers.c
+++ b/tests/qtest/migration-helpers.c
@@ -17,10 +17,12 @@
 
 bool got_stop;
 
-static void stop_cb(void *opaque, const char *name, QDict *data)
+static void check_stop_event(QTestState *who)
 {
-if (!strcmp(name, "STOP")) {
+QDict *event = qtest_qmp_event_ref(who, "STOP");
+if (event) {
 got_stop = true;
+qobject_unref(event);
 }
 }
 
@@ -30,12 +32,19 @@ static void stop_cb(void *opaque, const char *name, QDict 
*data)
 QDict *wait_command_fd(QTestState *who, int fd, const char *command, ...)
 {
 va_list ap;
+QDict *resp;
 
 va_start(ap, command);
 qtest_qm

[PATCH V2 4/5] target/riscv: Add V extension state description

[Yifei Jiang]

In the case of supporting V extension, add V extension description
to vmstate_riscv_cpu.

Signed-off-by: Yifei Jiang 
Signed-off-by: Yipeng Yin 
Reviewed-by: Richard Henderson 
---
 target/riscv/machine.c | 25 +
 1 file changed, 25 insertions(+)

diff --git a/target/riscv/machine.c b/target/riscv/machine.c
index af599bf001..2bd3a7b5b0 100644
--- a/target/riscv/machine.c
+++ b/target/riscv/machine.c
@@ -76,6 +76,30 @@ static bool hyper_needed(void *opaque)
 return riscv_has_ext(env, RVH);
 }
 
+static bool vector_needed(void *opaque)
+{
+RISCVCPU *cpu = opaque;
+CPURISCVState *env = &cpu->env;
+
+return riscv_has_ext(env, RVV);
+}
+
+static const VMStateDescription vmstate_vector = {
+.name = "cpu/vector",
+.version_id = 1,
+.minimum_version_id = 1,
+.needed = vector_needed,
+.fields = (VMStateField[]) {
+VMSTATE_UINT64_ARRAY(env.vreg, RISCVCPU, 32 * RV_VLEN_MAX / 64),
+VMSTATE_UINTTL(env.vxrm, RISCVCPU),
+VMSTATE_UINTTL(env.vxsat, RISCVCPU),
+VMSTATE_UINTTL(env.vl, RISCVCPU),
+VMSTATE_UINTTL(env.vstart, RISCVCPU),
+VMSTATE_UINTTL(env.vtype, RISCVCPU),
+VMSTATE_END_OF_LIST()
+}
+};
+
 static const VMStateDescription vmstate_hyper = {
 .name = "cpu/hyper",
 .version_id = 1,
@@ -173,6 +197,7 @@ const VMStateDescription vmstate_riscv_cpu = {
 .subsections = (const VMStateDescription * []) {
 &vmstate_pmp,
 &vmstate_hyper,
+&vmstate_vector,
 NULL
 }
 };
-- 
2.19.1

Re: [PATCH V13 1/9] linux-headers: Update MIPS KVM type defintition

[Philippe Mathieu-Daudé]

On 10/7/20 10:39 AM, Huacai Chen wrote:

Update MIPS KVM type defintition from Linux 5.9-rc6.


Reviewed-by: Philippe Mathieu-Daudé 



Signed-off-by: Huacai Chen 
---
  linux-headers/linux/kvm.h | 5 +++--
  1 file changed, 3 insertions(+), 2 deletions(-)

diff --git a/linux-headers/linux/kvm.h b/linux-headers/linux/kvm.h
index 6683e2e..c138b2f 100644
--- a/linux-headers/linux/kvm.h
+++ b/linux-headers/linux/kvm.h
@@ -790,9 +790,10 @@ struct kvm_ppc_resize_hpt {
  #define KVM_VM_PPC_HV 1
  #define KVM_VM_PPC_PR 2
  
-/* on MIPS, 0 forces trap & emulate, 1 forces VZ ASE */

-#define KVM_VM_MIPS_TE 0
+/* on MIPS, 0 indicates auto, 1 forces VZ ASE, 2 forces trap & emulate */
+#define KVM_VM_MIPS_AUTO   0
  #define KVM_VM_MIPS_VZ1
+#define KVM_VM_MIPS_TE 2
  
  #define KVM_S390_SIE_PAGE_OFFSET 1
  

[PATCH V2 2/5] target/riscv: Add PMP state description

[Yifei Jiang]

In the case of supporting PMP feature, add PMP state description
to vmstate_riscv_cpu.

'vmstate_pmp_addr' and 'num_rules' could be regenerated by
pmp_update_rule(). But there exists the problem of updating
num_rules repeatedly in pmp_update_rule(). So here extracts
pmp_update_rule_addr() and pmp_update_rule_nums() to update
'vmstate_pmp_addr' and 'num_rules' respectively.

Signed-off-by: Yifei Jiang 
Signed-off-by: Yipeng Yin 
---
 target/riscv/machine.c | 50 ++
 target/riscv/pmp.c | 29 ++--
 target/riscv/pmp.h |  2 ++
 3 files changed, 70 insertions(+), 11 deletions(-)

diff --git a/target/riscv/machine.c b/target/riscv/machine.c
index af2828a80a..8fffd2db3f 100644
--- a/target/riscv/machine.c
+++ b/target/riscv/machine.c
@@ -22,6 +22,52 @@
 #include "sysemu/kvm.h"
 #include "migration/cpu.h"
 
+static bool pmp_needed(void *opaque)
+{
+RISCVCPU *cpu = opaque;
+CPURISCVState *env = &cpu->env;
+
+return riscv_feature(env, RISCV_FEATURE_PMP);
+}
+
+static int pmp_post_load(void *opaque, int version_id)
+{
+RISCVCPU *cpu = opaque;
+CPURISCVState *env = &cpu->env;
+int i;
+
+for (i = 0; i < MAX_RISCV_PMPS; i++) {
+pmp_update_rule_addr(env, i);
+}
+pmp_update_rule_nums(env);
+
+return 0;
+}
+
+static const VMStateDescription vmstate_pmp_entry = {
+.name = "cpu/pmp/entry",
+.version_id = 1,
+.minimum_version_id = 1,
+.fields = (VMStateField[]) {
+VMSTATE_UINTTL(addr_reg, pmp_entry_t),
+VMSTATE_UINT8(cfg_reg, pmp_entry_t),
+VMSTATE_END_OF_LIST()
+}
+};
+
+static const VMStateDescription vmstate_pmp = {
+.name = "cpu/pmp",
+.version_id = 1,
+.minimum_version_id = 1,
+.needed = pmp_needed,
+.post_load = pmp_post_load,
+.fields = (VMStateField[]) {
+VMSTATE_STRUCT_ARRAY(env.pmp_state.pmp, RISCVCPU, MAX_RISCV_PMPS,
+ 0, vmstate_pmp_entry, pmp_entry_t),
+VMSTATE_END_OF_LIST()
+}
+};
+
 const VMStateDescription vmstate_riscv_cpu = {
 .name = "cpu",
 .version_id = 1,
@@ -73,5 +119,9 @@ const VMStateDescription vmstate_riscv_cpu = {
 VMSTATE_UINTTL(env.mstatush, RISCVCPU),
 #endif
 VMSTATE_END_OF_LIST()
+},
+.subsections = (const VMStateDescription * []) {
+&vmstate_pmp,
+NULL
 }
 };
diff --git a/target/riscv/pmp.c b/target/riscv/pmp.c
index c394e867f8..2eda8e1e2f 100644
--- a/target/riscv/pmp.c
+++ b/target/riscv/pmp.c
@@ -136,18 +136,8 @@ static void pmp_decode_napot(target_ulong a, target_ulong 
*sa, target_ulong *ea)
 }
 }
 
-
-/* Convert cfg/addr reg values here into simple 'sa' --> start address and 'ea'
- *   end address values.
- *   This function is called relatively infrequently whereas the check that
- *   an address is within a pmp rule is called often, so optimise that one
- */
-static void pmp_update_rule(CPURISCVState *env, uint32_t pmp_index)
+void pmp_update_rule_addr(CPURISCVState *env, uint32_t pmp_index)
 {
-int i;
-
-env->pmp_state.num_rules = 0;
-
 uint8_t this_cfg = env->pmp_state.pmp[pmp_index].cfg_reg;
 target_ulong this_addr = env->pmp_state.pmp[pmp_index].addr_reg;
 target_ulong prev_addr = 0u;
@@ -186,7 +176,13 @@ static void pmp_update_rule(CPURISCVState *env, uint32_t 
pmp_index)
 
 env->pmp_state.addr[pmp_index].sa = sa;
 env->pmp_state.addr[pmp_index].ea = ea;
+}
 
+void pmp_update_rule_nums(CPURISCVState *env)
+{
+int i;
+
+env->pmp_state.num_rules = 0;
 for (i = 0; i < MAX_RISCV_PMPS; i++) {
 const uint8_t a_field =
 pmp_get_a_field(env->pmp_state.pmp[i].cfg_reg);
@@ -196,6 +192,17 @@ static void pmp_update_rule(CPURISCVState *env, uint32_t 
pmp_index)
 }
 }
 
+/* Convert cfg/addr reg values here into simple 'sa' --> start address and 'ea'
+ *   end address values.
+ *   This function is called relatively infrequently whereas the check that
+ *   an address is within a pmp rule is called often, so optimise that one
+ */
+static void pmp_update_rule(CPURISCVState *env, uint32_t pmp_index)
+{
+pmp_update_rule_addr(env, pmp_index);
+pmp_update_rule_nums(env);
+}
+
 static int pmp_is_in_range(CPURISCVState *env, int pmp_index, target_ulong 
addr)
 {
 int result = 0;
diff --git a/target/riscv/pmp.h b/target/riscv/pmp.h
index 6a8f072871..6c6b4c9bef 100644
--- a/target/riscv/pmp.h
+++ b/target/riscv/pmp.h
@@ -62,5 +62,7 @@ bool pmp_hart_has_privs(CPURISCVState *env, target_ulong addr,
 target_ulong size, pmp_priv_t priv, target_ulong mode);
 bool pmp_is_range_in_tlb(CPURISCVState *env, hwaddr tlb_sa,
  target_ulong *tlb_size);
+void pmp_update_rule_addr(CPURISCVState *env, uint32_t pmp_index);
+void pmp_update_rule_nums(CPURISCVState *env);
 
 #endif
-- 
2.19.1

[PULL 33/39] scsi: switch to bus->check_address

[Paolo Bonzini]

Signed-off-by: Paolo Bonzini 
Message-Id: <20201006123904.610658-6-mlevi...@redhat.com>
Signed-off-by: Paolo Bonzini 
---
 hw/scsi/scsi-bus.c | 122 -
 1 file changed, 75 insertions(+), 47 deletions(-)

diff --git a/hw/scsi/scsi-bus.c b/hw/scsi/scsi-bus.c
index 4cf1f404b4..4ab9811cd8 100644
--- a/hw/scsi/scsi-bus.c
+++ b/hw/scsi/scsi-bus.c
@@ -22,33 +22,6 @@ static void scsi_req_dequeue(SCSIRequest *req);
 static uint8_t *scsi_target_alloc_buf(SCSIRequest *req, size_t len);
 static void scsi_target_free_buf(SCSIRequest *req);
 
-static Property scsi_props[] = {
-DEFINE_PROP_UINT32("channel", SCSIDevice, channel, 0),
-DEFINE_PROP_UINT32("scsi-id", SCSIDevice, id, -1),
-DEFINE_PROP_UINT32("lun", SCSIDevice, lun, -1),
-DEFINE_PROP_END_OF_LIST(),
-};
-
-static void scsi_bus_class_init(ObjectClass *klass, void *data)
-{
-BusClass *k = BUS_CLASS(klass);
-HotplugHandlerClass *hc = HOTPLUG_HANDLER_CLASS(klass);
-
-k->get_dev_path = scsibus_get_dev_path;
-k->get_fw_dev_path = scsibus_get_fw_dev_path;
-hc->unplug = qdev_simple_device_unplug_cb;
-}
-
-static const TypeInfo scsi_bus_info = {
-.name = TYPE_SCSI_BUS,
-.parent = TYPE_BUS,
-.instance_size = sizeof(SCSIBus),
-.class_init = scsi_bus_class_init,
-.interfaces = (InterfaceInfo[]) {
-{ TYPE_HOTPLUG_HANDLER },
-{ }
-}
-};
 static int next_scsi_bus;
 
 static void scsi_device_realize(SCSIDevice *s, Error **errp)
@@ -160,35 +133,68 @@ static void scsi_dma_restart_cb(void *opaque, int 
running, RunState state)
 }
 }
 
-static void scsi_qdev_realize(DeviceState *qdev, Error **errp)
+static bool scsi_bus_is_address_free(SCSIBus *bus,
+int channel, int target, int lun,
+SCSIDevice **p_dev)
+{
+SCSIDevice *d = scsi_device_find(bus, channel, target, lun);
+if (d && d->lun == lun) {
+if (p_dev) {
+*p_dev = d;
+}
+return false;
+}
+if (p_dev) {
+*p_dev = NULL;
+}
+return true;
+}
+
+static bool scsi_bus_check_address(BusState *qbus, DeviceState *qdev, Error 
**errp)
 {
 SCSIDevice *dev = SCSI_DEVICE(qdev);
-SCSIBus *bus = DO_UPCAST(SCSIBus, qbus, dev->qdev.parent_bus);
-SCSIDevice *d;
-Error *local_err = NULL;
+SCSIBus *bus = SCSI_BUS(qbus);
 
 if (dev->channel > bus->info->max_channel) {
 error_setg(errp, "bad scsi channel id: %d", dev->channel);
-return;
+return false;
 }
 if (dev->id != -1 && dev->id > bus->info->max_target) {
 error_setg(errp, "bad scsi device id: %d", dev->id);
-return;
+return false;
 }
 if (dev->lun != -1 && dev->lun > bus->info->max_lun) {
 error_setg(errp, "bad scsi device lun: %d", dev->lun);
-return;
+return false;
+}
+
+if (dev->id != -1 && dev->lun != -1) {
+SCSIDevice *d;
+if (!scsi_bus_is_address_free(bus, dev->channel, dev->id, dev->lun, 
&d)) {
+error_setg(errp, "lun already used by '%s'", d->qdev.id);
+return false;
+}
 }
 
+return true;
+}
+
+static void scsi_qdev_realize(DeviceState *qdev, Error **errp)
+{
+SCSIDevice *dev = SCSI_DEVICE(qdev);
+SCSIBus *bus = DO_UPCAST(SCSIBus, qbus, dev->qdev.parent_bus);
+bool is_free;
+Error *local_err = NULL;
+
 if (dev->id == -1) {
 int id = -1;
 if (dev->lun == -1) {
 dev->lun = 0;
 }
 do {
-d = scsi_device_find(bus, dev->channel, ++id, dev->lun);
-} while (d && d->lun == dev->lun && id < bus->info->max_target);
-if (d && d->lun == dev->lun) {
+is_free = scsi_bus_is_address_free(bus, dev->channel, ++id, 
dev->lun, NULL);
+} while (!is_free && id < bus->info->max_target);
+if (!is_free) {
 error_setg(errp, "no free target");
 return;
 }
@@ -196,20 +202,13 @@ static void scsi_qdev_realize(DeviceState *qdev, Error 
**errp)
 } else if (dev->lun == -1) {
 int lun = -1;
 do {
-d = scsi_device_find(bus, dev->channel, dev->id, ++lun);
-} while (d && d->lun == lun && lun < bus->info->max_lun);
-if (d && d->lun == lun) {
+is_free = scsi_bus_is_address_free(bus, dev->channel, dev->id, 
++lun, NULL);
+} while (!is_free && lun < bus->info->max_lun);
+if (!is_free) {
 error_setg(errp, "no free lun");
 return;
 }
 dev->lun = lun;
-} else {
-d = scsi_device_find(bus, dev->channel, dev->id, dev->lun);
-assert(d);
-if (d->lun == dev->lun && dev != d) {
-error_setg(errp, "lun already used by '%s'", d->qdev.id);
-return;
-}
 }
 
 QTAILQ_INIT(&dev->requests);
@@ -1723,6 +1722,13 @@ const VMStateDescription vmstate_scsi_device = {
 }
 };
 
+static Proper

[PATCH] docs/devel/testing.rst: Update outdated Avocado URLs

[Philippe Mathieu-Daudé]

Avocado documentation referred returns 404 error.
Update the broken links.

Signed-off-by: Philippe Mathieu-Daudé 
---
 docs/devel/testing.rst | 4 ++--
 1 file changed, 2 insertions(+), 2 deletions(-)

diff --git a/docs/devel/testing.rst b/docs/devel/testing.rst
index bd64c1bdcdd..23a1697d9f4 100644
--- a/docs/devel/testing.rst
+++ b/docs/devel/testing.rst
@@ -696,7 +696,7 @@ To manually install Avocado and its dependencies, run:
 
 Alternatively, follow the instructions on this link:
 
-  
http://avocado-framework.readthedocs.io/en/latest/GetStartedGuide.html#installing-avocado
+  
https://avocado-framework.readthedocs.io/en/latest/guides/user/chapters/installing.html
 
 Overview
 
@@ -879,7 +879,7 @@ Parameter reference
 To understand how Avocado parameters are accessed by tests, and how
 they can be passed to tests, please refer to::
 
-  
http://avocado-framework.readthedocs.io/en/latest/WritingTests.html#accessing-test-parameters
+  
https://avocado-framework.readthedocs.io/en/latest/guides/writer/chapters/writing.html#accessing-test-parameters
 
 Parameter values can be easily seen in the log files, and will look
 like the following:
-- 
2.26.2

[PULL 31/39] device_core: use drain_call_rcu in in qmp_device_add

[Paolo Bonzini]

From: Maxim Levitsky 

Soon, a device removal might only happen on RCU callback execution.
This is okay for device-del which provides a DEVICE_DELETED event,
but not for the failure case of device-add.  To avoid changing
monitor semantics, just drain all pending RCU callbacks on error.

Signed-off-by: Maxim Levitsky 
Suggested-by: Stefan Hajnoczi 
Reviewed-by: Stefan Hajnoczi 
Message-Id: <20200913160259.32145-4-mlevi...@redhat.com>
[Don't use it in qmp_device_del. - Paolo]
Signed-off-by: Paolo Bonzini 
---
 softmmu/qdev-monitor.c | 12 
 1 file changed, 12 insertions(+)

diff --git a/softmmu/qdev-monitor.c b/softmmu/qdev-monitor.c
index e9b7228480..bcfb90a08f 100644
--- a/softmmu/qdev-monitor.c
+++ b/softmmu/qdev-monitor.c
@@ -803,6 +803,18 @@ void qmp_device_add(QDict *qdict, QObject **ret_data, 
Error **errp)
 return;
 }
 dev = qdev_device_add(opts, errp);
+
+/*
+ * Drain all pending RCU callbacks. This is done because
+ * some bus related operations can delay a device removal
+ * (in this case this can happen if device is added and then
+ * removed due to a configuration error)
+ * to a RCU callback, but user might expect that this interface
+ * will finish its job completely once qmp command returns result
+ * to the user
+ */
+drain_call_rcu();
+
 if (!dev) {
 qemu_opts_del(opts);
 return;
-- 
2.26.2

[PULL 37/39] virtio-scsi: use scsi_device_get

[Paolo Bonzini]

From: Maxim Levitsky 

This will help us to avoid the scsi device disappearing
after we took a reference to it.

It doesn't by itself forbid case when we try to access
an unrealized device

Suggested-by: Stefan Hajnoczi 
Signed-off-by: Maxim Levitsky 
Reviewed-by: Stefan Hajnoczi 
Message-Id: <20200913160259.32145-9-mlevi...@redhat.com>
Signed-off-by: Paolo Bonzini 
Message-Id: <20201006123904.610658-13-mlevi...@redhat.com>
Signed-off-by: Paolo Bonzini 
---
 hw/scsi/virtio-scsi.c | 21 +
 1 file changed, 13 insertions(+), 8 deletions(-)

diff --git a/hw/scsi/virtio-scsi.c b/hw/scsi/virtio-scsi.c
index 971afbb217..3db9a8aae9 100644
--- a/hw/scsi/virtio-scsi.c
+++ b/hw/scsi/virtio-scsi.c
@@ -33,7 +33,7 @@ static inline int virtio_scsi_get_lun(uint8_t *lun)
 return ((lun[2] << 8) | lun[3]) & 0x3FFF;
 }
 
-static inline SCSIDevice *virtio_scsi_device_find(VirtIOSCSI *s, uint8_t *lun)
+static inline SCSIDevice *virtio_scsi_device_get(VirtIOSCSI *s, uint8_t *lun)
 {
 if (lun[0] != 1) {
 return NULL;
@@ -41,7 +41,7 @@ static inline SCSIDevice *virtio_scsi_device_find(VirtIOSCSI 
*s, uint8_t *lun)
 if (lun[2] != 0 && !(lun[2] >= 0x40 && lun[2] < 0x80)) {
 return NULL;
 }
-return scsi_device_find(&s->bus, 0, lun[1], virtio_scsi_get_lun(lun));
+return scsi_device_get(&s->bus, 0, lun[1], virtio_scsi_get_lun(lun));
 }
 
 void virtio_scsi_init_req(VirtIOSCSI *s, VirtQueue *vq, VirtIOSCSIReq *req)
@@ -256,7 +256,7 @@ static inline void virtio_scsi_ctx_check(VirtIOSCSI *s, 
SCSIDevice *d)
  *  case of async cancellation. */
 static int virtio_scsi_do_tmf(VirtIOSCSI *s, VirtIOSCSIReq *req)
 {
-SCSIDevice *d = virtio_scsi_device_find(s, req->req.tmf.lun);
+SCSIDevice *d = virtio_scsi_device_get(s, req->req.tmf.lun);
 SCSIRequest *r, *next;
 BusChild *kid;
 int target;
@@ -370,10 +370,10 @@ static int virtio_scsi_do_tmf(VirtIOSCSI *s, 
VirtIOSCSIReq *req)
 
 rcu_read_lock();
 QTAILQ_FOREACH_RCU(kid, &s->bus.qbus.children, sibling) {
- d = SCSI_DEVICE(kid->child);
- if (d->channel == 0 && d->id == target) {
-qdev_reset_all(&d->qdev);
- }
+SCSIDevice *d1 = SCSI_DEVICE(kid->child);
+if (d1->channel == 0 && d1->id == target) {
+qdev_reset_all(&d1->qdev);
+}
 }
 rcu_read_unlock();
 
@@ -386,14 +386,17 @@ static int virtio_scsi_do_tmf(VirtIOSCSI *s, 
VirtIOSCSIReq *req)
 break;
 }
 
+object_unref(OBJECT(d));
 return ret;
 
 incorrect_lun:
 req->resp.tmf.response = VIRTIO_SCSI_S_INCORRECT_LUN;
+object_unref(OBJECT(d));
 return ret;
 
 fail:
 req->resp.tmf.response = VIRTIO_SCSI_S_BAD_TARGET;
+object_unref(OBJECT(d));
 return ret;
 }
 
@@ -564,7 +567,7 @@ static int virtio_scsi_handle_cmd_req_prepare(VirtIOSCSI 
*s, VirtIOSCSIReq *req)
 }
 }
 
-d = virtio_scsi_device_find(s, req->req.cmd.lun);
+d = virtio_scsi_device_get(s, req->req.cmd.lun);
 if (!d) {
 req->resp.cmd.response = VIRTIO_SCSI_S_BAD_TARGET;
 virtio_scsi_complete_cmd_req(req);
@@ -580,10 +583,12 @@ static int virtio_scsi_handle_cmd_req_prepare(VirtIOSCSI 
*s, VirtIOSCSIReq *req)
 req->sreq->cmd.xfer > req->qsgl.size)) {
 req->resp.cmd.response = VIRTIO_SCSI_S_OVERRUN;
 virtio_scsi_complete_cmd_req(req);
+object_unref(OBJECT(d));
 return -ENOBUFS;
 }
 scsi_req_ref(req->sreq);
 blk_io_plug(d->conf.blk);
+object_unref(OBJECT(d));
 return 0;
 }
 
-- 
2.26.2

[PATCH V2 3/5] target/riscv: Add H extension state description

[Yifei Jiang]

In the case of supporting H extension, add H extension description
to vmstate_riscv_cpu.

Signed-off-by: Yifei Jiang 
Signed-off-by: Yipeng Yin 
---
 target/riscv/machine.c | 51 ++
 1 file changed, 51 insertions(+)

diff --git a/target/riscv/machine.c b/target/riscv/machine.c
index 8fffd2db3f..af599bf001 100644
--- a/target/riscv/machine.c
+++ b/target/riscv/machine.c
@@ -68,6 +68,56 @@ static const VMStateDescription vmstate_pmp = {
 }
 };
 
+static bool hyper_needed(void *opaque)
+{
+RISCVCPU *cpu = opaque;
+CPURISCVState *env = &cpu->env;
+
+return riscv_has_ext(env, RVH);
+}
+
+static const VMStateDescription vmstate_hyper = {
+.name = "cpu/hyper",
+.version_id = 1,
+.minimum_version_id = 1,
+.needed = hyper_needed,
+.fields = (VMStateField[]) {
+VMSTATE_UINTTL(env.hstatus, RISCVCPU),
+VMSTATE_UINTTL(env.hedeleg, RISCVCPU),
+VMSTATE_UINTTL(env.hideleg, RISCVCPU),
+VMSTATE_UINTTL(env.hcounteren, RISCVCPU),
+VMSTATE_UINTTL(env.htval, RISCVCPU),
+VMSTATE_UINTTL(env.htinst, RISCVCPU),
+VMSTATE_UINTTL(env.hgatp, RISCVCPU),
+VMSTATE_UINT64(env.htimedelta, RISCVCPU),
+
+VMSTATE_UINTTL(env.vsstatus, RISCVCPU),
+VMSTATE_UINTTL(env.vstvec, RISCVCPU),
+VMSTATE_UINTTL(env.vsscratch, RISCVCPU),
+VMSTATE_UINTTL(env.vsepc, RISCVCPU),
+VMSTATE_UINTTL(env.vscause, RISCVCPU),
+VMSTATE_UINTTL(env.vstval, RISCVCPU),
+VMSTATE_UINTTL(env.vsatp, RISCVCPU),
+
+VMSTATE_UINTTL(env.mtval2, RISCVCPU),
+VMSTATE_UINTTL(env.mtinst, RISCVCPU),
+
+VMSTATE_UINTTL(env.stvec_hs, RISCVCPU),
+VMSTATE_UINTTL(env.sscratch_hs, RISCVCPU),
+VMSTATE_UINTTL(env.sepc_hs, RISCVCPU),
+VMSTATE_UINTTL(env.scause_hs, RISCVCPU),
+VMSTATE_UINTTL(env.stval_hs, RISCVCPU),
+VMSTATE_UINTTL(env.satp_hs, RISCVCPU),
+VMSTATE_UINTTL(env.mstatus_hs, RISCVCPU),
+
+#ifdef TARGET_RISCV32
+VMSTATE_UINTTL(env.vsstatush, RISCVCPU),
+VMSTATE_UINTTL(env.mstatush_hs, RISCVCPU),
+#endif
+VMSTATE_END_OF_LIST()
+}
+};
+
 const VMStateDescription vmstate_riscv_cpu = {
 .name = "cpu",
 .version_id = 1,
@@ -122,6 +172,7 @@ const VMStateDescription vmstate_riscv_cpu = {
 },
 .subsections = (const VMStateDescription * []) {
 &vmstate_pmp,
+&vmstate_hyper,
 NULL
 }
 };
-- 
2.19.1

Re: [PULL 00/39] SCSI, qdev, qtest, meson patches for 2020-10-10

[no-reply]

Patchew URL: 
https://patchew.org/QEMU/20201010075739.951385-1-pbonz...@redhat.com/



Hi,

This series seems to have some coding style problems. See output below for
more information:

Type: series
Message-id: 20201010075739.951385-1-pbonz...@redhat.com
Subject: [PULL 00/39] SCSI, qdev, qtest, meson patches for 2020-10-10

=== TEST SCRIPT BEGIN ===
#!/bin/bash
git rev-parse base > /dev/null || exit 0
git config --local diff.renamelimit 0
git config --local diff.renames True
git config --local diff.algorithm histogram
./scripts/checkpatch.pl --mailback base..
=== TEST SCRIPT END ===

Updating 3c8cf5a9c21ff8782164d1def7f44bd888713384
From https://github.com/patchew-project/qemu
 - [tag update]  
patchew/1602059975-10115-1-git-send-email-che...@lemote.com -> 
patchew/1602059975-10115-1-git-send-email-che...@lemote.com
 * [new tag] patchew/20201010075739.951385-1-pbonz...@redhat.com -> 
patchew/20201010075739.951385-1-pbonz...@redhat.com
 * [new tag] patchew/20201010080623.768-1-jiangyi...@huawei.com -> 
patchew/20201010080623.768-1-jiangyi...@huawei.com
Switched to a new branch 'test'
55f98d2 meson: identify more sections of meson.build
3f959b1 scsi/scsi_bus: fix races in REPORT LUNS
41c759b virtio-scsi: use scsi_device_get
17f4dca scsi/scsi_bus: Add scsi_device_get
eafe5c7 scsi/scsi-bus: scsi_device_find: don't return unrealized devices
c8ddbc5 device-core: use atomic_set on .realized property
07df212 scsi: switch to bus->check_address
a7baa24 device-core: use RCU for list of children of a bus
bbd9350 device_core: use drain_call_rcu in in qmp_device_add
7cafd84 scsi/scsi_bus: switch search direction in scsi_device_find
9ef1542 qdev: add "check if address free" callback for buses
1d6727d qemu-iotests, qtest: rewrite test 067 as a qtest
96b23e6 qtest: check that drives are really appearing and disappearing
a6d906c qtest: switch users back to qtest_qmp_receive
fcf3581 device-plug-test: use qtest_qmp to send the device_del command
3722046 qtest: remove qtest_qmp_receive_success
17077b6 qtest: Reintroduce qtest_qmp_receive
5b850fc qtest: rename qtest_qmp_receive to qtest_qmp_receive_dict
940c32f meson.build: Re-enable KVM support for MIPS
b4cd071 build-sys: fix git version from -version
e81db1e docs/devel: update instruction on how to add new unit tests
f187a57 qtest: unify extra_qtest_srcs and extra_qtest_deps
80c356b docs/devel/qtest: Include libqtest API reference
ef6b5cb docs/devel/qtest: Include protocol spec in document
7f82ce7 docs: Move QTest documentation to its own document
d5fddc0 configure: fix performance regression due to PIC objects
70cd0b9 qom: fix objects with improper parent type
a63dc48 exec: split out non-softmmu-specific parts
6a633f6 softmmu: move more files to softmmu/
32c750b hw/nvram: Always register FW_CFG_DATA_GENERATOR_INTERFACE
85a528c qom: Move the creation of the library to the main meson.build
6d611b1 authz: Move the creation of the library to the main meson.build
d597f26 crypto: Move the creation of the library to the main meson.build
d44bc2e io: Move the creation of the library to the main meson.build
6f014b0 migration: Move the creation of the library to the main meson.build
9d197f1 chardev: Move the creation of the library to the main meson.build
8b9a449 hw/core: Move the creation of the library to the main meson.build
245f27b meson.build: Sort sourcesets alphabetically
cbc9573 meson.build: Add comments to clarify code organization

=== OUTPUT BEGIN ===
1/39 Checking commit cbc957397c02 (meson.build: Add comments to clarify code 
organization)
2/39 Checking commit 245f27b24fa7 (meson.build: Sort sourcesets alphabetically)
3/39 Checking commit 8b9a44915aa8 (hw/core: Move the creation of the library to 
the main meson.build)
4/39 Checking commit 9d197f154484 (chardev: Move the creation of the library to 
the main meson.build)
5/39 Checking commit 6f014b047dde (migration: Move the creation of the library 
to the main meson.build)
6/39 Checking commit d44bc2e97f1c (io: Move the creation of the library to the 
main meson.build)
7/39 Checking commit d597f26a6c94 (crypto: Move the creation of the library to 
the main meson.build)
8/39 Checking commit 6d611b137d38 (authz: Move the creation of the library to 
the main meson.build)
9/39 Checking commit 85a528c294eb (qom: Move the creation of the library to the 
main meson.build)
10/39 Checking commit 32c750b90a9c (hw/nvram: Always register 
FW_CFG_DATA_GENERATOR_INTERFACE)
WARNING: added, moved or deleted file(s), does MAINTAINERS need updating?
#52: 
new file mode 100644

total: 0 errors, 1 warnings, 55 lines checked

Patch 10/39 has style problems, please review.  If any of these errors
are false positives report them to the maintainer, see
CHECKPATCH in MAINTAINERS.
11/39 Checking commit 6a633f68ddd7 (softmmu: move more files to softmmu/)
WARNING: added, moved or deleted file(s), does MAINTAINERS need updating?
#85: 
rename from bootdevice.c

total: 0 errors, 1 warnings, 69 lines checked

Patch 11/39 has style pro

Re: [PATCH v10 0/7] vhost-user block device backend implementation

[Coiby Xu]

On Fri, Oct 09, 2020 at 11:18:43AM +0100, Stefan Hajnoczi wrote:

On Fri, Sep 18, 2020 at 04:09:05PM +0800, Coiby Xu wrote:

v10
 - Use a linked list of VuFdWatch objects to keep kick info [Stefan]
 - Merge improvements and fixes from Stefan
 - Rename libvhost-user's vu_message_read to vu_message_read_default [Kevin]
 - When shutting down the client, wait for the coroutine of processing
   vhost-user messages to be finished [Kevin]
 - Allocate struct req_data on the heap [Kevin]
 - Improve coding of qtest case [Thomas]
 - Fix several memory leaks detected by ASAN

v9
 - move logical block size check function to a utility function
 - fix issues regarding license, coding style, memory deallocation, etc.

v8
 - re-try connecting to socket server to fix asan error
 - fix license naming issue

v7
 - fix docker-test-debug@fedora errors by freeing malloced memory

v6
 - add missing license header and include guard
 - vhost-user server only serve one client one time
 - fix a bug in custom vu_message_read
 - using qemu-storage-daemon to start vhost-user-blk-server
 - a bug fix to pass docker-test-clang@ubuntu

v5:
 * re-use vu_kick_cb in libvhost-user
 * keeping processing VhostUserMsg in the same coroutine until there is
   detachment/attachment of AIOContext
 * Spawn separate coroutine for each VuVirtqElement
 * Other changes including relocating vhost-user-blk-server.c, coding
   style etc.

v4:
 * add object properties in class_init
 * relocate vhost-user-blk-test
 * other changes including using SocketAddress, coding style, etc.

v3:
 * separate generic vhost-user-server code from vhost-user-blk-server
   code
 * re-write vu_message_read and kick hander function as coroutines to
   directly call blk_co_preadv, blk_co_pwritev, etc.
 * add aio_context notifier functions to support multi-threading model
 * other fixes regarding coding style, warning report, etc.

v2:
 * Only enable this feature for Linux because eventfd is a Linux-specific
   feature


This patch series is an implementation of vhost-user block device
backend server, thanks to Stefan and Kevin's guidance.

Vhost-user block device backend server is a UserCreatable object and can be
started using object_add,

 (qemu) object_add 
vhost-user-blk-server,id=ID,unix-socket=/tmp/vhost-user-blk_vhost.socket,node-name=DRIVE_NAME,writable=off,logical-block-size=512
 (qemu) object_del ID

or appending the "-object" option when starting QEMU,

  $ -object 
vhost-user-blk-server,id=disk,unix-socket=/tmp/vhost-user-blk_vhost.socket,node-name=DRIVE_NAME,writable=off,logical-block-size=512

Then vhost-user client can connect to the server backend.
For example, QEMU could act as a client,

  $ -m 256 -object memory-backend-memfd,id=mem,size=256M,share=on -numa 
node,memdev=mem -chardev socket,id=char1,path=/tmp/vhost-user-blk_vhost.socket 
-device vhost-user-blk-pci,id=blk0,chardev=char1

And guest OS could access this vhost-user block device after mounting it.

Coiby Xu (7):
  libvhost-user: Allow vu_message_read to be replaced
  libvhost-user: remove watch for kick_fd when de-initialize vu-dev
  util/vhost-user-server: generic vhost user server
  block: move logical block size check function to a common utility
function
  block/export: vhost-user block device backend server
  test: new qTest case to test the vhost-user-blk-server
  MAINTAINERS: Add vhost-user block device backend server maintainer

 MAINTAINERS|   8 +
 block/export/vhost-user-blk-server.c   | 661 ++
 block/export/vhost-user-blk-server.h   |  36 +
 block/meson.build  |   1 +
 contrib/libvhost-user/libvhost-user-glib.c |   2 +-
 contrib/libvhost-user/libvhost-user.c  |  15 +-
 contrib/libvhost-user/libvhost-user.h  |  21 +
 hw/core/qdev-properties.c  |  31 +-
 softmmu/vl.c   |   4 +
 tests/qtest/libqos/libqtest.h  |  17 +
 tests/qtest/libqos/meson.build |   1 +
 tests/qtest/libqos/vhost-user-blk.c| 129 
 tests/qtest/libqos/vhost-user-blk.h|  48 ++
 tests/qtest/libqtest.c |  36 +-
 tests/qtest/meson.build|   4 +-
 tests/qtest/vhost-user-blk-test.c  | 751 +
 tests/vhost-user-bridge.c  |   2 +
 tools/virtiofsd/fuse_virtio.c  |   4 +-
 util/block-helpers.c   |  46 ++
 util/block-helpers.h   |  19 +
 util/meson.build   |   2 +
 util/vhost-user-server.c   | 428 
 util/vhost-user-server.h   |  65 ++
 23 files changed, 2292 insertions(+), 39 deletions(-)
 create mode 100644 block/export/vhost-user-blk-server.c
 create mode 100644 block/export/vhost-user-blk-server.h
 create mode 100644 tests/qtest/libqos/vhost-user-blk.c
 create mode 100644 tests/qtest/libqos/vhost-user-blk.h
 create mode 100644 tests/qtest/vhost-user-blk-test.c

Re: [PATCH V13 8/9] hw/mips: Add Loongson-3 machine support

[Philippe Mathieu-Daudé]

Hi Huacai,

On 10/7/20 10:39 AM, Huacai Chen wrote:

Add Loongson-3 based machine support, it use liointc as the interrupt
controler and use GPEX as the pci controller. Currently it can work with
both TCG and KVM.

As the machine model is not based on any exiting physical hardware, the
name of the machine is "loongson3-virt". It may be superseded in future
by a real machine model. If this happens, then a regular deprecation
procedure shall occur for "loongson3-virt" machine.

We now already have a full functional Linux kernel (based on Linux-5.4.x
LTS, the kvm host side and guest side have both been upstream for Linux-
5.9, but Linux-5.9 has not been released yet) here:

https://github.com/chenhuacai/linux

Of course the upstream kernel is also usable (though it is "unstable"
now):

https://git.kernel.org/pub/scm/linux/kernel/git/torvalds/linux.git

How to use QEMU/Loongson-3?
1, Download kernel source from the above URL;
2, Build a kernel with arch/mips/configs/loongson3_defconfig;
3, Boot a Loongson-3A4000 host with this kernel (for KVM mode);
4, Build QEMU-master with this patchset;
5, modprobe kvm (only necessary for KVM mode);
6, Use QEMU with TCG:
qemu-system-mips64el -M loongson3-virt,accel=tcg -cpu Loongson-3A1000 -kernel 
 -append ...
Use QEMU with KVM:
qemu-system-mips64el -M loongson3-virt,accel=kvm -cpu Loongson-3A4000 -kernel 
 -append ...

The "-cpu" parameter is optional here and QEMU will use the correct type 
for TCG/KVM automatically.

Signed-off-by: Huacai Chen 
Co-developed-by: Jiaxun Yang 
---
  default-configs/devices/mips64el-softmmu.mak |   1 +
  hw/mips/Kconfig  |  12 +
  hw/mips/loongson3_bootp.c| 162 +++
  hw/mips/loongson3_bootp.h| 225 ++


In a previous version I asked if you could split the bootp* files
(firmware related) in a previous separate patch to ease the review
of the machine part.


  hw/mips/loongson3_virt.c | 615 +++
  hw/mips/meson.build  |   1 +
  6 files changed, 1016 insertions(+)

[...]


diff --git a/hw/mips/loongson3_virt.c b/hw/mips/loongson3_virt.c
new file mode 100644
index 000..4e573d6
--- /dev/null
+++ b/hw/mips/loongson3_virt.c
@@ -0,0 +1,615 @@
+/*
+ * Generic Loongson-3 Platform support
+ *
+ * Copyright (c) 2017-2020 Huacai Chen (che...@lemote.com)
+ * Copyright (c) 2017-2020 Jiaxun Yang 
+ *
+ * This program is free software: you can redistribute it and/or modify
+ * it under the terms of the GNU General Public License as published by
+ * the Free Software Foundation, either version 2 of the License, or
+ * (at your option) any later version.
+ *
+ * This program is distributed in the hope that it will be useful,
+ * but WITHOUT ANY WARRANTY; without even the implied warranty of
+ * MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the
+ * GNU General Public License for more details.
+ *
+ * You should have received a copy of the GNU General Public License
+ * along with this program. If not, see .
+ */
+
+/*
+ * Generic virtualized PC Platform based on Loongson-3 CPU (MIPS64R2 with
+ * extensions, 800~2000MHz)
+ */
+
+#include "qemu/osdep.h"
+#include "qemu-common.h"
+#include "qemu/units.h"
+#include "qemu/cutils.h"
+#include "qapi/error.h"
+#include "cpu.h"
+#include "elf.h"
+#include "kvm_mips.h"
+#include "hw/boards.h"
+#include "hw/char/serial.h"
+#include "hw/mips/mips.h"
+#include "hw/mips/cpudevs.h"
+#include "hw/mips/fw_cfg.h"
+#include "hw/mips/loongson3_bootp.h"
+#include "hw/misc/unimp.h"
+#include "hw/intc/i8259.h"
+#include "hw/loader.h"
+#include "hw/isa/superio.h"
+#include "hw/pci/msi.h"
+#include "hw/pci/pci.h"
+#include "hw/pci/pci_host.h"
+#include "hw/pci-host/gpex.h"
+#include "hw/rtc/mc146818rtc.h"
+#include "hw/usb.h"
+#include "net/net.h"
+#include "exec/address-spaces.h"
+#include "sysemu/kvm.h"
+#include "sysemu/qtest.h"
+#include "sysemu/reset.h"
+#include "sysemu/runstate.h"
+#include "qemu/log.h"
+#include "qemu/error-report.h"
+
+#define PM_CNTL_MODE  0x10
+
+#define LOONGSON_MAX_VCPUS  16
+
+#define LOONGSON3_BIOSNAME "bios_loongson3.bin"


Where is that file?


+
+#define UART_IRQ0
+#define RTC_IRQ 1
+#define PCIE_IRQ_BASE   2
+
+const struct MemmapEntry virt_memmap[] = {
+[VIRT_LOWMEM] =  { 0x,0x1000 },
+[VIRT_PM] =  { 0x1008, 0x100 },
+[VIRT_FW_CFG] =  { 0x10080100, 0x100 },
+[VIRT_RTC] = { 0x10081000,0x1000 },
+[VIRT_PCIE_PIO] ={ 0x1800,   0x8 },
+[VIRT_PCIE_ECAM] =   { 0x1a00, 0x200 },
+[VIRT_BIOS_ROM] ={ 0x1fc0,  0x20 },
+[VIRT_UART] ={ 0x1fe001e0,   0x8 },


See below about this mapping.


+[VIRT_LIOINTC] = { 0x3ff01400,  0x64 },
+[VIRT_PCIE_MMIO] =   { 0x4000,0x4000 },
+

Re: [PATCH 2/3] Acceptance tests: do not show canceled test logs on GitLab CI

[Philippe Mathieu-Daudé]

On 10/9/20 10:55 PM, Cleber Rosa wrote:

Tests resulting in "CANCEL" in Avocado are usually canceled on
purpose, and are almost identical to "SKIP".  The logs for canceled
tests are adding a lot of noise to the logs being shown on GitLab CI,
and causing distraction from real failures.


Thanks, big improvement!!!

Reviewed-by: Philippe Mathieu-Daudé 
Tested-by: Philippe Mathieu-Daudé 



As a side note, this "after script" is scheduled for removal once the
feature is implemented within Avocado itself.

Reference: https://github.com/avocado-framework/avocado/issues/4266
Signed-off-by: Cleber Rosa 
---
  .gitlab-ci.yml | 2 +-
  1 file changed, 1 insertion(+), 1 deletion(-)

diff --git a/.gitlab-ci.yml b/.gitlab-ci.yml
index a51c89554f..bed5fe6161 100644
--- a/.gitlab-ci.yml
+++ b/.gitlab-ci.yml
@@ -63,7 +63,7 @@ include:
fi
after_script:
  - cd build
-- python3 -c 'import json; r = json.load(open("tests/results/latest/results.json")); [print(t["logfile"]) for t in 
r["tests"] if t["status"] not in ("PASS", "SKIP")]' | xargs cat
+- python3 -c 'import json; r = json.load(open("tests/results/latest/results.json")); [print(t["logfile"]) for t in 
r["tests"] if t["status"] not in ("PASS", "SKIP", "CANCEL")]' | xargs cat
  - du -chs ${CI_PROJECT_DIR}/avocado-cache
  
  build-system-ubuntu:

Re: [PATCH 3/3] Acceptance tests: show test report on GitLab CI

[Philippe Mathieu-Daudé]

On 10/9/20 10:55 PM, Cleber Rosa wrote:

Avocado will, by default, produce JUnit files.  Let's ask GitLab
to present those in the web UI.


https://gitlab.com/philmd/qemu/-/pipelines/200764992/test_report

Reviewed-by: Philippe Mathieu-Daudé 
Tested-by: Philippe Mathieu-Daudé 



Signed-off-by: Cleber Rosa 
---
  .gitlab-ci.yml | 5 +
  1 file changed, 5 insertions(+)

diff --git a/.gitlab-ci.yml b/.gitlab-ci.yml
index bed5fe6161..7d4b2ced9e 100644
--- a/.gitlab-ci.yml
+++ b/.gitlab-ci.yml
@@ -53,6 +53,11 @@ include:
  paths:
- ${CI_PROJECT_DIR}/avocado-cache
  policy: pull-push
+  artifacts:
+paths:
+  - build/tests/results/latest/results.xml
+reports:
+  junit: build/tests/results/latest/results.xml
before_script:
  - mkdir -p ~/.config/avocado
  - echo "[datadir.paths]" > ~/.config/avocado/avocado.conf

Re: [PATCH v6 00/36] qapi: static typing conversion, pt1

[Markus Armbruster]

John Snow  writes:

> Hi, this series adds static type hints to the QAPI module.
> This is part one!
>
> Part 1: https://gitlab.com/jsnow/qemu/-/tree/python-qapi-cleanup-pt1
> Everything: https://gitlab.com/jsnow/qemu/-/tree/python-qapi-cleanup-pt6
>
> - Requires Python 3.6+
> - Requires mypy 0.770 or newer (for type analysis only)
> - Requires pylint 2.6.0 or newer (for lint checking only)
>
> In general, this series tackles the cleanup of one individual QAPI
> module at a time. Once it passes pylint or mypy checks, those checks are
> enabled for that file.
>
> Type hints are added in patches that add *only* type hints and change no
> other behavior. Any necessary changes to behavior to accommodate typing
> are split out into their own tiny patches.
>
> Notes:
>
> - After patch 07, `isort -c` should pass 100% on this and every
>   future commit.
>
> - After patch 08, `flake8 qapi/` should pass 100% on this and every
>   future commit.
>
> - After patch 09, `pylint --rcfile=qapi/pylintrc qapi/` should pass 100%
>   on this and every future commit.
>
> - After patch 18, `mypy --config-file=qapi/mypy.ini qapi/` should pass
>   100% on this and every future commit.

Series
Reviewed-by: Markus Armbruster 

Queued, thanks!

[PULL 01/34] docs: repair broken references

[Markus Armbruster]

From: John Snow 

In two different places, we are not making a cross-reference to some
resource correctly.

Signed-off-by: John Snow 
Message-Id: <20201009161558.107041-2-js...@redhat.com>
Reviewed-by: Markus Armbruster 
Signed-off-by: Markus Armbruster 
---
 docs/devel/multi-thread-tcg.rst | 2 +-
 docs/devel/testing.rst  | 2 +-
 2 files changed, 2 insertions(+), 2 deletions(-)

diff --git a/docs/devel/multi-thread-tcg.rst b/docs/devel/multi-thread-tcg.rst
index 21483870db..92a9eba13c 100644
--- a/docs/devel/multi-thread-tcg.rst
+++ b/docs/devel/multi-thread-tcg.rst
@@ -267,7 +267,7 @@ of view of external observers (e.g. another processor 
core). They can
 apply to any memory operations as well as just loads or stores.
 
 The Linux kernel has an excellent `write-up
-`
+`_
 on the various forms of memory barrier and the guarantees they can
 provide.
 
diff --git a/docs/devel/testing.rst b/docs/devel/testing.rst
index bd64c1bdcd..8875a40a2b 100644
--- a/docs/devel/testing.rst
+++ b/docs/devel/testing.rst
@@ -953,7 +953,7 @@ compiler flags are needed to build for a given target.
 If you have the ability to run containers as the user you can also
 take advantage of the build systems "Docker" support. It will then use
 containers to build any test case for an enabled guest where there is
-no system compiler available. See :ref: `_docker-ref` for details.
+no system compiler available. See :ref:`docker-ref` for details.
 
 Running subset of tests
 ---
-- 
2.26.2

[PULL 11/34] qapi/common.py: Add indent manager

[Markus Armbruster]

From: John Snow 

Code style tools really dislike the use of global keywords, because it
generally involves re-binding the name at runtime which can have strange
effects depending on when and how that global name is referenced in
other modules.

Make a little indent level manager instead.

Signed-off-by: John Snow 
Reviewed-by: Eduardo Habkost 
Reviewed-by: Cleber Rosa 
Message-Id: <20201009161558.107041-12-js...@redhat.com>
Reviewed-by: Markus Armbruster 
Signed-off-by: Markus Armbruster 
---
 scripts/qapi/common.py | 57 +++---
 scripts/qapi/visit.py  |  7 +++---
 2 files changed, 40 insertions(+), 24 deletions(-)

diff --git a/scripts/qapi/common.py b/scripts/qapi/common.py
index cee63eb95c..b35318b72c 100644
--- a/scripts/qapi/common.py
+++ b/scripts/qapi/common.py
@@ -93,33 +93,50 @@ eatspace = '\033EATSPACE.'
 pointer_suffix = ' *' + eatspace
 
 
-def genindent(count):
-ret = ''
-for _ in range(count):
-ret += ' '
-return ret
+class Indentation:
+"""
+Indentation level management.
+
+:param initial: Initial number of spaces, default 0.
+"""
+def __init__(self, initial: int = 0) -> None:
+self._level = initial
+
+def __int__(self) -> int:
+return self._level
+
+def __repr__(self) -> str:
+return "{}({:d})".format(type(self).__name__, self._level)
+
+def __str__(self) -> str:
+"""Return the current indentation as a string of spaces."""
+return ' ' * self._level
+
+def __bool__(self) -> bool:
+"""True when there is a non-zero indentation."""
+return bool(self._level)
+
+def increase(self, amount: int = 4) -> None:
+"""Increase the indentation level by ``amount``, default 4."""
+self._level += amount
+
+def decrease(self, amount: int = 4) -> None:
+"""Decrease the indentation level by ``amount``, default 4."""
+if self._level < amount:
+raise ArithmeticError(
+f"Can't remove {amount:d} spaces from {self!r}")
+self._level -= amount
 
 
-indent_level = 0
-
-
-def push_indent(indent_amount=4):
-global indent_level
-indent_level += indent_amount
-
-
-def pop_indent(indent_amount=4):
-global indent_level
-indent_level -= indent_amount
+indent = Indentation()
 
 
 # Generate @code with @kwds interpolated.
-# Obey indent_level, and strip eatspace.
+# Obey indent, and strip eatspace.
 def cgen(code, **kwds):
 raw = code % kwds
-if indent_level:
-indent = genindent(indent_level)
-raw = re.sub(r'^(?!(#|$))', indent, raw, flags=re.MULTILINE)
+if indent:
+raw = re.sub(r'^(?!(#|$))', str(indent), raw, flags=re.MULTILINE)
 return re.sub(re.escape(eatspace) + r' *', '', raw)
 
 
diff --git a/scripts/qapi/visit.py b/scripts/qapi/visit.py
index 9fdbe5b9ef..708f72c4a1 100644
--- a/scripts/qapi/visit.py
+++ b/scripts/qapi/visit.py
@@ -18,9 +18,8 @@ from .common import (
 c_name,
 gen_endif,
 gen_if,
+indent,
 mcgen,
-pop_indent,
-push_indent,
 )
 from .gen import QAPISchemaModularCVisitor, ifcontext
 from .schema import QAPISchemaObjectType
@@ -69,7 +68,7 @@ bool visit_type_%(c_name)s_members(Visitor *v, %(c_name)s 
*obj, Error **errp)
 if (visit_optional(v, "%(name)s", &obj->has_%(c_name)s)) {
 ''',
  name=memb.name, c_name=c_name(memb.name))
-push_indent()
+indent.increase()
 ret += mcgen('''
 if (!visit_type_%(c_type)s(v, "%(name)s", &obj->%(c_name)s, errp)) {
 return false;
@@ -78,7 +77,7 @@ bool visit_type_%(c_name)s_members(Visitor *v, %(c_name)s 
*obj, Error **errp)
  c_type=memb.type.c_name(), name=memb.name,
  c_name=c_name(memb.name))
 if memb.optional:
-pop_indent()
+indent.decrease()
 ret += mcgen('''
 }
 ''')
-- 
2.26.2

[PULL 03/34] qapi-gen: Separate arg-parsing from generation

[Markus Armbruster]

From: John Snow 

This is a minor re-work of the entrypoint script. It isolates a
generate() method from the actual command-line mechanism.

Signed-off-by: John Snow 
Reviewed-by: Eduardo Habkost 
Reviewed-by: Cleber Rosa 
Tested-by: Cleber Rosa 
Message-Id: <20201009161558.107041-4-js...@redhat.com>
Reviewed-by: Markus Armbruster 
[invalid_char() renamed to invalid_prefix_char()]
Signed-off-by: Markus Armbruster 
---
 scripts/qapi-gen.py | 89 ++---
 1 file changed, 67 insertions(+), 22 deletions(-)

diff --git a/scripts/qapi-gen.py b/scripts/qapi-gen.py
index 541e8c1f55..cf942d9ce3 100644
--- a/scripts/qapi-gen.py
+++ b/scripts/qapi-gen.py
@@ -1,30 +1,77 @@
 #!/usr/bin/env python3
-# QAPI generator
-#
+
 # This work is licensed under the terms of the GNU GPL, version 2 or later.
 # See the COPYING file in the top-level directory.
 
+"""
+QAPI Generator
+
+This is the main entry point for generating C code from the QAPI schema.
+"""
 
 import argparse
 import re
 import sys
+from typing import Optional
 
 from qapi.commands import gen_commands
+from qapi.error import QAPIError
 from qapi.events import gen_events
 from qapi.introspect import gen_introspect
-from qapi.schema import QAPIError, QAPISchema
+from qapi.schema import QAPISchema
 from qapi.types import gen_types
 from qapi.visit import gen_visit
 
 
-def main(argv):
+def invalid_prefix_char(prefix: str) -> Optional[str]:
+match = re.match(r'([A-Za-z_.-][A-Za-z0-9_.-]*)?', prefix)
+if match.end() != len(prefix):
+return prefix[match.end()]
+return None
+
+
+def generate(schema_file: str,
+ output_dir: str,
+ prefix: str,
+ unmask: bool = False,
+ builtins: bool = False) -> None:
+"""
+Generate C code for the given schema into the target directory.
+
+:param schema_file: The primary QAPI schema file.
+:param output_dir: The output directory to store generated code.
+:param prefix: Optional C-code prefix for symbol names.
+:param unmask: Expose non-ABI names through introspection?
+:param builtins: Generate code for built-in types?
+
+:raise QAPIError: On failures.
+"""
+assert invalid_prefix_char(prefix) is None
+
+schema = QAPISchema(schema_file)
+gen_types(schema, output_dir, prefix, builtins)
+gen_visit(schema, output_dir, prefix, builtins)
+gen_commands(schema, output_dir, prefix)
+gen_events(schema, output_dir, prefix)
+gen_introspect(schema, output_dir, prefix, unmask)
+
+
+def main() -> int:
+"""
+gapi-gen executable entry point.
+Expects arguments via sys.argv, see --help for details.
+
+:return: int, 0 on success, 1 on failure.
+"""
 parser = argparse.ArgumentParser(
 description='Generate code from a QAPI schema')
 parser.add_argument('-b', '--builtins', action='store_true',
 help="generate code for built-in types")
-parser.add_argument('-o', '--output-dir', action='store', default='',
+parser.add_argument('-o', '--output-dir', action='store',
+default='',
 help="write output to directory OUTPUT_DIR")
-parser.add_argument('-p', '--prefix', action='store', default='',
+parser.add_argument('-p', '--prefix', action='store',
+default='',
 help="prefix for symbols")
 parser.add_argument('-u', '--unmask-non-abi-names', action='store_true',
 dest='unmask',
@@ -32,25 +79,23 @@ def main(argv):
 parser.add_argument('schema', action='store')
 args = parser.parse_args()
 
-match = re.match(r'([A-Za-z_.-][A-Za-z0-9_.-]*)?', args.prefix)
-if match.end() != len(args.prefix):
-print("%s: 'funny character '%s' in argument of --prefix"
-  % (sys.argv[0], args.prefix[match.end()]),
-  file=sys.stderr)
-sys.exit(1)
+funny_char = invalid_prefix_char(args.prefix)
+if funny_char:
+msg = f"funny character '{funny_char}' in argument of --prefix"
+print(f"{sys.argv[0]}: {msg}", file=sys.stderr)
+return 1
 
 try:
-schema = QAPISchema(args.schema)
+generate(args.schema,
+ output_dir=args.output_dir,
+ prefix=args.prefix,
+ unmask=args.unmask,
+ builtins=args.builtins)
 except QAPIError as err:
-print(err, file=sys.stderr)
-exit(1)
-
-gen_types(schema, args.output_dir, args.prefix, args.builtins)
-gen_visit(schema, args.output_dir, args.prefix, args.builtins)
-gen_commands(schema, args.output_dir, args.prefix)
-gen_events(schema, args.output_dir, args.prefix)
-gen_introspect(schema, args.output_dir, args.prefix, args.unmask)
+print(f"{sys.argv[0]}: {str(err)}", file=sys.stderr)
+return 1
+return 0
 
 
 if __name__ == '__main__':
-main(sys.argv)
+sys.exit(main())
-- 
2.

[PULL 13/34] qapi/common.py: Replace one-letter 'c' variable

[Markus Armbruster]

From: John Snow 

Signed-off-by: John Snow 
Reviewed-by: Cleber Rosa 
Reviewed-by: Eduardo Habkost 
Reviewed-by: Philippe Mathieu-Daudé 
Message-Id: <20201009161558.107041-14-js...@redhat.com>
Reviewed-by: Markus Armbruster 
Signed-off-by: Markus Armbruster 
---
 scripts/qapi/common.py | 8 
 1 file changed, 4 insertions(+), 4 deletions(-)

diff --git a/scripts/qapi/common.py b/scripts/qapi/common.py
index a417b6029c..338adedef4 100644
--- a/scripts/qapi/common.py
+++ b/scripts/qapi/common.py
@@ -30,14 +30,14 @@ def camel_to_upper(value):
 new_name = ''
 length = len(c_fun_str)
 for i in range(length):
-c = c_fun_str[i]
-# When c is upper and no '_' appears before, do more checks
-if c.isupper() and (i > 0) and c_fun_str[i - 1] != '_':
+char = c_fun_str[i]
+# When char is upper case and no '_' appears before, do more checks
+if char.isupper() and (i > 0) and c_fun_str[i - 1] != '_':
 if i < length - 1 and c_fun_str[i + 1].islower():
 new_name += '_'
 elif c_fun_str[i - 1].isdigit():
 new_name += '_'
-new_name += c
+new_name += char
 return new_name.lstrip('_').upper()
 
 
-- 
2.26.2

[PULL 00/34] QAPI patches patches for 2020-10-10

[Markus Armbruster]

The following changes since commit 4a7c0bd9dcb08798c6f82e55b5a3423f7ee669f1:

  Merge remote-tracking branch 'remotes/dgibson/tags/ppc-for-5.2-20201009' into 
staging (2020-10-09 15:48:04 +0100)

are available in the Git repository at:

  git://repo.or.cz/qemu/armbru.git tags/pull-qapi-2020-10-10

for you to fetch changes up to b4c0aa59aff520e2a55edd5fef393058ca6520de:

  qapi/visit.py: add type hint annotations (2020-10-10 11:37:49 +0200)


QAPI patches patches for 2020-10-10


John Snow (34):
  docs: repair broken references
  qapi: modify docstrings to be sphinx-compatible
  qapi-gen: Separate arg-parsing from generation
  qapi: move generator entrypoint into package
  qapi: Prefer explicit relative imports
  qapi: Remove wildcard includes
  qapi: enforce import order/styling with isort
  qapi: delint using flake8
  qapi: add pylintrc
  qapi/common.py: Remove python compatibility workaround
  qapi/common.py: Add indent manager
  qapi/common.py: delint with pylint
  qapi/common.py: Replace one-letter 'c' variable
  qapi/common.py: check with pylint
  qapi/common.py: add type hint annotations
  qapi/common.py: Convert comments into docstrings, and elaborate
  qapi/common.py: move build_params into gen.py
  qapi: establish mypy type-checking baseline
  qapi/events.py: add type hint annotations
  qapi/events.py: Move comments into docstrings
  qapi/commands.py: Don't re-bind to variable of different type
  qapi/commands.py: add type hint annotations
  qapi/source.py: add type hint annotations
  qapi/source.py: delint with pylint
  qapi/gen: Make _is_user_module() return bool
  qapi/gen.py: add type hint annotations
  qapi/gen.py: Remove unused parameter
  qapi/gen.py: update write() to be more idiomatic
  qapi/gen.py: delint with pylint
  qapi/types.py: add type hint annotations
  qapi/types.py: remove one-letter variables
  qapi/visit.py: assert tag_member contains a QAPISchemaEnumType
  qapi/visit.py: remove unused parameters from gen_visit_object
  qapi/visit.py: add type hint annotations

 docs/devel/multi-thread-tcg.rst |   2 +-
 docs/devel/testing.rst  |   2 +-
 scripts/qapi-gen.py |  57 +++--
 scripts/qapi/.flake8|   2 +
 scripts/qapi/.isort.cfg |   7 ++
 scripts/qapi/commands.py|  90 ++--
 scripts/qapi/common.py  | 174 +-
 scripts/qapi/events.py  |  58 +
 scripts/qapi/expr.py|   7 +-
 scripts/qapi/gen.py | 180 +---
 scripts/qapi/introspect.py  |  16 +++-
 scripts/qapi/main.py|  95 +
 scripts/qapi/mypy.ini   |  30 +++
 scripts/qapi/parser.py  |   6 +-
 scripts/qapi/pylintrc   |  70 
 scripts/qapi/schema.py  |  33 
 scripts/qapi/source.py  |  35 +---
 scripts/qapi/types.py   | 125 +++-
 scripts/qapi/visit.py   | 116 +++---
 19 files changed, 764 insertions(+), 341 deletions(-)
 create mode 100644 scripts/qapi/.flake8
 create mode 100644 scripts/qapi/.isort.cfg
 create mode 100644 scripts/qapi/main.py
 create mode 100644 scripts/qapi/mypy.ini
 create mode 100644 scripts/qapi/pylintrc

-- 
2.26.2

[PULL 06/34] qapi: Remove wildcard includes

[Markus Armbruster]

From: John Snow 

Wildcard includes become hard to manage when refactoring and dealing
with circular dependencies with strictly typed mypy.

flake8 also flags each one as a warning, as it is not smart enough to
know which names exist in the imported file.

Remove them and include things explicitly by name instead.

Signed-off-by: John Snow 
Reviewed-by: Eduardo Habkost 
Reviewed-by: Cleber Rosa 
Reviewed-by: Philippe Mathieu-Daudé 
Message-Id: <20201009161558.107041-7-js...@redhat.com>
Reviewed-by: Markus Armbruster 
Signed-off-by: Markus Armbruster 
---
 scripts/qapi/commands.py   |  2 +-
 scripts/qapi/events.py |  7 ++-
 scripts/qapi/gen.py| 12 +---
 scripts/qapi/introspect.py |  7 ++-
 scripts/qapi/types.py  |  8 +++-
 scripts/qapi/visit.py  | 10 +-
 6 files changed, 38 insertions(+), 8 deletions(-)

diff --git a/scripts/qapi/commands.py b/scripts/qapi/commands.py
index 1f43a0a34e..e06c10afcd 100644
--- a/scripts/qapi/commands.py
+++ b/scripts/qapi/commands.py
@@ -13,7 +13,7 @@ This work is licensed under the terms of the GNU GPL, version 
2.
 See the COPYING file in the top-level directory.
 """
 
-from .common import *
+from .common import build_params, c_name, mcgen
 from .gen import QAPIGenCCode, QAPISchemaModularCVisitor, ifcontext
 
 
diff --git a/scripts/qapi/events.py b/scripts/qapi/events.py
index 0467272438..6b3afa14d7 100644
--- a/scripts/qapi/events.py
+++ b/scripts/qapi/events.py
@@ -12,7 +12,12 @@ This work is licensed under the terms of the GNU GPL, 
version 2.
 See the COPYING file in the top-level directory.
 """
 
-from .common import *
+from .common import (
+build_params,
+c_enum_const,
+c_name,
+mcgen,
+)
 from .gen import QAPISchemaModularCVisitor, ifcontext
 from .schema import QAPISchemaEnumMember
 from .types import gen_enum, gen_enum_lookup
diff --git a/scripts/qapi/gen.py b/scripts/qapi/gen.py
index fc57fdca5b..1fed712b43 100644
--- a/scripts/qapi/gen.py
+++ b/scripts/qapi/gen.py
@@ -11,13 +11,19 @@
 # This work is licensed under the terms of the GNU GPL, version 2.
 # See the COPYING file in the top-level directory.
 
-
+from contextlib import contextmanager
 import errno
 import os
 import re
-from contextlib import contextmanager
 
-from .common import *
+from .common import (
+c_fname,
+gen_endif,
+gen_if,
+guardend,
+guardstart,
+mcgen,
+)
 from .schema import QAPISchemaVisitor
 
 
diff --git a/scripts/qapi/introspect.py b/scripts/qapi/introspect.py
index 6c82d9d95f..42016a7e66 100644
--- a/scripts/qapi/introspect.py
+++ b/scripts/qapi/introspect.py
@@ -10,7 +10,12 @@ This work is licensed under the terms of the GNU GPL, 
version 2.
 See the COPYING file in the top-level directory.
 """
 
-from .common import *
+from .common import (
+c_name,
+gen_endif,
+gen_if,
+mcgen,
+)
 from .gen import QAPISchemaMonolithicCVisitor
 from .schema import (QAPISchemaArrayType, QAPISchemaBuiltinType,
  QAPISchemaType)
diff --git a/scripts/qapi/types.py b/scripts/qapi/types.py
index ca9a5aacb3..53b47f9e58 100644
--- a/scripts/qapi/types.py
+++ b/scripts/qapi/types.py
@@ -13,7 +13,13 @@ This work is licensed under the terms of the GNU GPL, 
version 2.
 # See the COPYING file in the top-level directory.
 """
 
-from .common import *
+from .common import (
+c_enum_const,
+c_name,
+gen_endif,
+gen_if,
+mcgen,
+)
 from .gen import QAPISchemaModularCVisitor, ifcontext
 from .schema import QAPISchemaEnumMember, QAPISchemaObjectType
 
diff --git a/scripts/qapi/visit.py b/scripts/qapi/visit.py
index 7850f6e848..ea277e7704 100644
--- a/scripts/qapi/visit.py
+++ b/scripts/qapi/visit.py
@@ -13,7 +13,15 @@ This work is licensed under the terms of the GNU GPL, 
version 2.
 See the COPYING file in the top-level directory.
 """
 
-from .common import *
+from .common import (
+c_enum_const,
+c_name,
+gen_endif,
+gen_if,
+mcgen,
+pop_indent,
+push_indent,
+)
 from .gen import QAPISchemaModularCVisitor, ifcontext
 from .schema import QAPISchemaObjectType
 
-- 
2.26.2

[PULL 10/34] qapi/common.py: Remove python compatibility workaround

[Markus Armbruster]

From: John Snow 

Signed-off-by: John Snow 
Reviewed-by: Eduardo Habkost 
Reviewed-by: Cleber Rosa 
Message-Id: <20201009161558.107041-11-js...@redhat.com>
Reviewed-by: Markus Armbruster 
Signed-off-by: Markus Armbruster 
---
 scripts/qapi/common.py | 5 +
 1 file changed, 1 insertion(+), 4 deletions(-)

diff --git a/scripts/qapi/common.py b/scripts/qapi/common.py
index ba35abea47..cee63eb95c 100644
--- a/scripts/qapi/common.py
+++ b/scripts/qapi/common.py
@@ -119,10 +119,7 @@ def cgen(code, **kwds):
 raw = code % kwds
 if indent_level:
 indent = genindent(indent_level)
-# re.subn() lacks flags support before Python 2.7, use re.compile()
-raw = re.subn(re.compile(r'^(?!(#|$))', re.MULTILINE),
-  indent, raw)
-raw = raw[0]
+raw = re.sub(r'^(?!(#|$))', indent, raw, flags=re.MULTILINE)
 return re.sub(re.escape(eatspace) + r' *', '', raw)
 
 
-- 
2.26.2

[PULL 04/34] qapi: move generator entrypoint into package

[Markus Armbruster]

From: John Snow 

As part of delinting and adding type hints to the QAPI generator, it's
helpful for the entrypoint to be part of the package, only leaving a
very tiny entrypoint shim outside of the package.

Signed-off-by: John Snow 
Reviewed-by: Eduardo Habkost 
Reviewed-by: Cleber Rosa 
Tested-by: Cleber Rosa 
Message-Id: <20201009161558.107041-5-js...@redhat.com>
Reviewed-by: Markus Armbruster 
[invalid_char() renamed to invalid_prefix_char()]
Signed-off-by: Markus Armbruster 
---
 scripts/qapi-gen.py  | 94 +++
 scripts/qapi/main.py | 95 
 2 files changed, 101 insertions(+), 88 deletions(-)
 create mode 100644 scripts/qapi/main.py

diff --git a/scripts/qapi-gen.py b/scripts/qapi-gen.py
index cf942d9ce3..f3518d29a5 100644
--- a/scripts/qapi-gen.py
+++ b/scripts/qapi-gen.py
@@ -4,98 +4,16 @@
 # See the COPYING file in the top-level directory.
 
 """
-QAPI Generator
+QAPI code generation execution shim.
 
-This is the main entry point for generating C code from the QAPI schema.
+This standalone script exists primarily to facilitate the running of the QAPI
+code generator without needing to install the python module to the current
+execution environment.
 """
 
-import argparse
-import re
 import sys
-from typing import Optional
-
-from qapi.commands import gen_commands
-from qapi.error import QAPIError
-from qapi.events import gen_events
-from qapi.introspect import gen_introspect
-from qapi.schema import QAPISchema
-from qapi.types import gen_types
-from qapi.visit import gen_visit
-
-
-def invalid_prefix_char(prefix: str) -> Optional[str]:
-match = re.match(r'([A-Za-z_.-][A-Za-z0-9_.-]*)?', prefix)
-if match.end() != len(prefix):
-return prefix[match.end()]
-return None
-
-
-def generate(schema_file: str,
- output_dir: str,
- prefix: str,
- unmask: bool = False,
- builtins: bool = False) -> None:
-"""
-Generate C code for the given schema into the target directory.
-
-:param schema_file: The primary QAPI schema file.
-:param output_dir: The output directory to store generated code.
-:param prefix: Optional C-code prefix for symbol names.
-:param unmask: Expose non-ABI names through introspection?
-:param builtins: Generate code for built-in types?
-
-:raise QAPIError: On failures.
-"""
-assert invalid_prefix_char(prefix) is None
-
-schema = QAPISchema(schema_file)
-gen_types(schema, output_dir, prefix, builtins)
-gen_visit(schema, output_dir, prefix, builtins)
-gen_commands(schema, output_dir, prefix)
-gen_events(schema, output_dir, prefix)
-gen_introspect(schema, output_dir, prefix, unmask)
-
-
-def main() -> int:
-"""
-gapi-gen executable entry point.
-Expects arguments via sys.argv, see --help for details.
-
-:return: int, 0 on success, 1 on failure.
-"""
-parser = argparse.ArgumentParser(
-description='Generate code from a QAPI schema')
-parser.add_argument('-b', '--builtins', action='store_true',
-help="generate code for built-in types")
-parser.add_argument('-o', '--output-dir', action='store',
-default='',
-help="write output to directory OUTPUT_DIR")
-parser.add_argument('-p', '--prefix', action='store',
-default='',
-help="prefix for symbols")
-parser.add_argument('-u', '--unmask-non-abi-names', action='store_true',
-dest='unmask',
-help="expose non-ABI names in introspection")
-parser.add_argument('schema', action='store')
-args = parser.parse_args()
-
-funny_char = invalid_prefix_char(args.prefix)
-if funny_char:
-msg = f"funny character '{funny_char}' in argument of --prefix"
-print(f"{sys.argv[0]}: {msg}", file=sys.stderr)
-return 1
-
-try:
-generate(args.schema,
- output_dir=args.output_dir,
- prefix=args.prefix,
- unmask=args.unmask,
- builtins=args.builtins)
-except QAPIError as err:
-print(f"{sys.argv[0]}: {str(err)}", file=sys.stderr)
-return 1
-return 0
 
+from qapi import main
 
 if __name__ == '__main__':
-sys.exit(main())
+sys.exit(main.main())
diff --git a/scripts/qapi/main.py b/scripts/qapi/main.py
new file mode 100644
index 00..92a2a31ca1
--- /dev/null
+++ b/scripts/qapi/main.py
@@ -0,0 +1,95 @@
+# This work is licensed under the terms of the GNU GPL, version 2 or later.
+# See the COPYING file in the top-level directory.
+
+"""
+QAPI Generator
+
+This is the main entry point for generating C code from the QAPI schema.
+"""
+
+import argparse
+import re
+import sys
+from typing import Optional
+
+from qapi.commands import gen_commands
+from qapi.error import QAPIError
+from qapi.events import gen_event

[PULL 07/34] qapi: enforce import order/styling with isort

[Markus Armbruster]

From: John Snow 

While we're mucking around with imports, we might as well formalize the
style we use. Let's use isort to do it for us.

lines_after_imports=2: Use two lines after imports, to match PEP8's
desire to have "two lines before and after" class definitions, which are
likely to start immediately after imports.

force_sort_within_sections: Intermingles "from x" and "import x" style
statements, such that sorting is always performed strictly on the module
name itself.

force_grid_wrap=4: Four or more imports from a single module will force
the one-per-line style that's more git-friendly. This will generally
happen for 'typing' imports.

multi_line_output=3: Uses the one-per-line indented style for long
imports.

include_trailing_comma: Adds a comma to the last import in a group,
which makes git conflicts nicer to deal with, generally.

line_length: 72 is chosen to match PEP8's "docstrings and comments" line
length limit. If you have a single line import that exceeds 72
characters, your names are too long!

Suggested-by: Cleber Rosa 
Signed-off-by: John Snow 
Reviewed-by: Cleber Rosa 
Tested-by: Cleber Rosa 
Reviewed-by: Markus Armbruster 
Message-Id: <20201009161558.107041-8-js...@redhat.com>
Signed-off-by: Markus Armbruster 
---
 scripts/qapi/.isort.cfg| 7 +++
 scripts/qapi/expr.py   | 3 ++-
 scripts/qapi/introspect.py | 7 +--
 scripts/qapi/parser.py | 2 +-
 scripts/qapi/schema.py | 2 +-
 5 files changed, 16 insertions(+), 5 deletions(-)
 create mode 100644 scripts/qapi/.isort.cfg

diff --git a/scripts/qapi/.isort.cfg b/scripts/qapi/.isort.cfg
new file mode 100644
index 00..643caa1fbd
--- /dev/null
+++ b/scripts/qapi/.isort.cfg
@@ -0,0 +1,7 @@
+[settings]
+force_grid_wrap=4
+force_sort_within_sections=True
+include_trailing_comma=True
+line_length=72
+lines_after_imports=2
+multi_line_output=3
diff --git a/scripts/qapi/expr.py b/scripts/qapi/expr.py
index bb4dc55f56..2fcaaa2497 100644
--- a/scripts/qapi/expr.py
+++ b/scripts/qapi/expr.py
@@ -14,8 +14,9 @@
 # This work is licensed under the terms of the GNU GPL, version 2.
 # See the COPYING file in the top-level directory.
 
-import re
 from collections import OrderedDict
+import re
+
 from .common import c_name
 from .error import QAPISemError
 
diff --git a/scripts/qapi/introspect.py b/scripts/qapi/introspect.py
index 42016a7e66..fafec94e02 100644
--- a/scripts/qapi/introspect.py
+++ b/scripts/qapi/introspect.py
@@ -17,8 +17,11 @@ from .common import (
 mcgen,
 )
 from .gen import QAPISchemaMonolithicCVisitor
-from .schema import (QAPISchemaArrayType, QAPISchemaBuiltinType,
- QAPISchemaType)
+from .schema import (
+QAPISchemaArrayType,
+QAPISchemaBuiltinType,
+QAPISchemaType,
+)
 
 
 def _make_tree(obj, ifcond, features, extra=None):
diff --git a/scripts/qapi/parser.py b/scripts/qapi/parser.py
index 7298f5dbd1..e7b9d670ad 100644
--- a/scripts/qapi/parser.py
+++ b/scripts/qapi/parser.py
@@ -14,9 +14,9 @@
 # This work is licensed under the terms of the GNU GPL, version 2.
 # See the COPYING file in the top-level directory.
 
+from collections import OrderedDict
 import os
 import re
-from collections import OrderedDict
 
 from .error import QAPIParseError, QAPISemError
 from .source import QAPISourceInfo
diff --git a/scripts/qapi/schema.py b/scripts/qapi/schema.py
index 676185d1a7..71ebb1e396 100644
--- a/scripts/qapi/schema.py
+++ b/scripts/qapi/schema.py
@@ -14,9 +14,9 @@
 
 # TODO catching name collisions in generated code would be nice
 
+from collections import OrderedDict
 import os
 import re
-from collections import OrderedDict
 
 from .common import c_name, pointer_suffix
 from .error import QAPIError, QAPISemError
-- 
2.26.2

[PULL 12/34] qapi/common.py: delint with pylint

[Markus Armbruster]

From: John Snow 

At this point, that just means using a consistent strategy for constant names.
constants get UPPER_CASE and names not used externally get a leading underscore.

As a preference, while renaming constants to be UPPERCASE, move them to
the head of the file. Generally, it's nice to be able to audit the code
that runs on import in one central place.

Signed-off-by: John Snow 
Reviewed-by: Eduardo Habkost 
Reviewed-by: Cleber Rosa 
Message-Id: <20201009161558.107041-13-js...@redhat.com>
Reviewed-by: Markus Armbruster 
Signed-off-by: Markus Armbruster 
---
 scripts/qapi/common.py | 18 --
 scripts/qapi/schema.py | 14 +++---
 2 files changed, 15 insertions(+), 17 deletions(-)

diff --git a/scripts/qapi/common.py b/scripts/qapi/common.py
index b35318b72c..a417b6029c 100644
--- a/scripts/qapi/common.py
+++ b/scripts/qapi/common.py
@@ -14,6 +14,11 @@
 import re
 
 
+EATSPACE = '\033EATSPACE.'
+POINTER_SUFFIX = ' *' + EATSPACE
+_C_NAME_TRANS = str.maketrans('.-', '__')
+
+
 # ENUMName -> ENUM_NAME, EnumName1 -> ENUM_NAME1
 # ENUM_NAME -> ENUM_NAME, ENUM_NAME1 -> ENUM_NAME1, ENUM_Name2 -> ENUM_NAME2
 # ENUM24_Name -> ENUM24_NAME
@@ -42,9 +47,6 @@ def c_enum_const(type_name, const_name, prefix=None):
 return camel_to_upper(type_name) + '_' + c_name(const_name, False).upper()
 
 
-c_name_trans = str.maketrans('.-', '__')
-
-
 # Map @name to a valid C identifier.
 # If @protect, avoid returning certain ticklish identifiers (like
 # C keywords) by prepending 'q_'.
@@ -82,17 +84,13 @@ def c_name(name, protect=True):
  'not_eq', 'or', 'or_eq', 'xor', 'xor_eq'])
 # namespace pollution:
 polluted_words = set(['unix', 'errno', 'mips', 'sparc', 'i386'])
-name = name.translate(c_name_trans)
+name = name.translate(_C_NAME_TRANS)
 if protect and (name in c89_words | c99_words | c11_words | gcc_words
 | cpp_words | polluted_words):
 return 'q_' + name
 return name
 
 
-eatspace = '\033EATSPACE.'
-pointer_suffix = ' *' + eatspace
-
-
 class Indentation:
 """
 Indentation level management.
@@ -132,12 +130,12 @@ indent = Indentation()
 
 
 # Generate @code with @kwds interpolated.
-# Obey indent, and strip eatspace.
+# Obey indent, and strip EATSPACE.
 def cgen(code, **kwds):
 raw = code % kwds
 if indent:
 raw = re.sub(r'^(?!(#|$))', str(indent), raw, flags=re.MULTILINE)
-return re.sub(re.escape(eatspace) + r' *', '', raw)
+return re.sub(re.escape(EATSPACE) + r' *', '', raw)
 
 
 def mcgen(code, **kwds):
diff --git a/scripts/qapi/schema.py b/scripts/qapi/schema.py
index afd750989e..7c01592956 100644
--- a/scripts/qapi/schema.py
+++ b/scripts/qapi/schema.py
@@ -18,7 +18,7 @@ from collections import OrderedDict
 import os
 import re
 
-from .common import c_name, pointer_suffix
+from .common import POINTER_SUFFIX, c_name
 from .error import QAPIError, QAPISemError
 from .expr import check_exprs
 from .parser import QAPISchemaParser
@@ -309,7 +309,7 @@ class QAPISchemaArrayType(QAPISchemaType):
 return True
 
 def c_type(self):
-return c_name(self.name) + pointer_suffix
+return c_name(self.name) + POINTER_SUFFIX
 
 def json_type(self):
 return 'array'
@@ -430,7 +430,7 @@ class QAPISchemaObjectType(QAPISchemaType):
 
 def c_type(self):
 assert not self.is_implicit()
-return c_name(self.name) + pointer_suffix
+return c_name(self.name) + POINTER_SUFFIX
 
 def c_unboxed_type(self):
 return c_name(self.name)
@@ -504,7 +504,7 @@ class QAPISchemaAlternateType(QAPISchemaType):
 v.connect_doc(doc)
 
 def c_type(self):
-return c_name(self.name) + pointer_suffix
+return c_name(self.name) + POINTER_SUFFIX
 
 def json_type(self):
 return 'value'
@@ -899,7 +899,7 @@ class QAPISchema:
 self._make_array_type(name, None)
 
 def _def_predefineds(self):
-for t in [('str','string',  'char' + pointer_suffix),
+for t in [('str','string',  'char' + POINTER_SUFFIX),
   ('number', 'number',  'double'),
   ('int','int', 'int64_t'),
   ('int8',   'int', 'int8_t'),
@@ -912,8 +912,8 @@ class QAPISchema:
   ('uint64', 'int', 'uint64_t'),
   ('size',   'int', 'uint64_t'),
   ('bool',   'boolean', 'bool'),
-  ('any','value',   'QObject' + pointer_suffix),
-  ('null',   'null','QNull' + pointer_suffix)]:
+  ('any','value',   'QObject' + POINTER_SUFFIX),
+  ('null',   'null','QNull' + POINTER_SUFFIX)]:
 self._def_builtin_type(*t)
 self.the_empty_object_type = QAPISchemaObjectType(
 'q_empty', None, None, None, None, None, [], None)
-- 
2.26.2

[PULL 05/34] qapi: Prefer explicit relative imports

[Markus Armbruster]

From: John Snow 

All of the QAPI include statements are changed to be package-aware, as
explicit relative imports.

A quirk of Python packages is that the name of the package exists only
*outside* of the package. This means that to a module inside of the qapi
folder, there is inherently no such thing as the "qapi" package. The
reason these imports work is because the "qapi" package exists in the
context of the caller -- the execution shim, where sys.path includes a
directory that has a 'qapi' folder in it.

When we write "from qapi import sibling", we are NOT referencing the folder
'qapi', but rather "any package named qapi in sys.path". If you should
so happen to have a 'qapi' package in your path, it will use *that*
package.

When we write "from .sibling import foo", we always reference explicitly
our sibling module; guaranteeing consistency in *where* we are importing
these modules from.

This can be useful when working with virtual environments and packages
in development mode. In development mode, a package is installed as a
series of symlinks that forwards to your same source files. The problem
arises because code quality checkers will follow "import qapi.x" to the
"installed" version instead of the sibling file and -- even though they
are the same file -- they have different module paths, and this causes
cyclic import problems, false positive type mismatch errors, and more.

It can also be useful when dealing with hierarchical packages, e.g. if
we allow qemu.core.qmp, qemu.qapi.parser, etc.

Signed-off-by: John Snow 
Reviewed-by: Eduardo Habkost 
Reviewed-by: Cleber Rosa 
Reviewed-by: Philippe Mathieu-Daudé 
Message-Id: <20201009161558.107041-6-js...@redhat.com>
Reviewed-by: Markus Armbruster 
Signed-off-by: Markus Armbruster 
---
 scripts/qapi/commands.py   |  4 ++--
 scripts/qapi/events.py |  8 
 scripts/qapi/expr.py   |  4 ++--
 scripts/qapi/gen.py|  4 ++--
 scripts/qapi/introspect.py |  8 
 scripts/qapi/main.py   | 14 +++---
 scripts/qapi/parser.py |  4 ++--
 scripts/qapi/schema.py |  8 
 scripts/qapi/types.py  |  6 +++---
 scripts/qapi/visit.py  |  6 +++---
 10 files changed, 33 insertions(+), 33 deletions(-)

diff --git a/scripts/qapi/commands.py b/scripts/qapi/commands.py
index 6e6fc94a14..1f43a0a34e 100644
--- a/scripts/qapi/commands.py
+++ b/scripts/qapi/commands.py
@@ -13,8 +13,8 @@ This work is licensed under the terms of the GNU GPL, version 
2.
 See the COPYING file in the top-level directory.
 """
 
-from qapi.common import *
-from qapi.gen import QAPIGenCCode, QAPISchemaModularCVisitor, ifcontext
+from .common import *
+from .gen import QAPIGenCCode, QAPISchemaModularCVisitor, ifcontext
 
 
 def gen_command_decl(name, arg_type, boxed, ret_type):
diff --git a/scripts/qapi/events.py b/scripts/qapi/events.py
index b544af5a1c..0467272438 100644
--- a/scripts/qapi/events.py
+++ b/scripts/qapi/events.py
@@ -12,10 +12,10 @@ This work is licensed under the terms of the GNU GPL, 
version 2.
 See the COPYING file in the top-level directory.
 """
 
-from qapi.common import *
-from qapi.gen import QAPISchemaModularCVisitor, ifcontext
-from qapi.schema import QAPISchemaEnumMember
-from qapi.types import gen_enum, gen_enum_lookup
+from .common import *
+from .gen import QAPISchemaModularCVisitor, ifcontext
+from .schema import QAPISchemaEnumMember
+from .types import gen_enum, gen_enum_lookup
 
 
 def build_event_send_proto(name, arg_type, boxed):
diff --git a/scripts/qapi/expr.py b/scripts/qapi/expr.py
index a15c1fb474..bb4dc55f56 100644
--- a/scripts/qapi/expr.py
+++ b/scripts/qapi/expr.py
@@ -16,8 +16,8 @@
 
 import re
 from collections import OrderedDict
-from qapi.common import c_name
-from qapi.error import QAPISemError
+from .common import c_name
+from .error import QAPISemError
 
 
 # Names must be letters, numbers, -, and _.  They must start with letter,
diff --git a/scripts/qapi/gen.py b/scripts/qapi/gen.py
index dc7b94aa11..fc57fdca5b 100644
--- a/scripts/qapi/gen.py
+++ b/scripts/qapi/gen.py
@@ -17,8 +17,8 @@ import os
 import re
 from contextlib import contextmanager
 
-from qapi.common import *
-from qapi.schema import QAPISchemaVisitor
+from .common import *
+from .schema import QAPISchemaVisitor
 
 
 class QAPIGen:
diff --git a/scripts/qapi/introspect.py b/scripts/qapi/introspect.py
index 5907b09cd5..6c82d9d95f 100644
--- a/scripts/qapi/introspect.py
+++ b/scripts/qapi/introspect.py
@@ -10,10 +10,10 @@ This work is licensed under the terms of the GNU GPL, 
version 2.
 See the COPYING file in the top-level directory.
 """
 
-from qapi.common import *
-from qapi.gen import QAPISchemaMonolithicCVisitor
-from qapi.schema import (QAPISchemaArrayType, QAPISchemaBuiltinType,
- QAPISchemaType)
+from .common import *
+from .gen import QAPISchemaMonolithicCVisitor
+from .schema import (QAPISchemaArrayType, QAPISchemaBuiltinType,
+ QAPISchemaType)
 
 
 def _make_tree(obj, ifcond, fe

[PULL 14/34] qapi/common.py: check with pylint

[Markus Armbruster]

From: John Snow 

Remove qapi/common.py from the pylintrc ignore list.

Signed-off-by: John Snow 
Reviewed-by: Cleber Rosa 
Tested-by: Cleber Rosa 
Tested-by: Eduardo Habkost 
Message-Id: <20201009161558.107041-15-js...@redhat.com>
Reviewed-by: Markus Armbruster 
Signed-off-by: Markus Armbruster 
---
 scripts/qapi/pylintrc | 3 +--
 1 file changed, 1 insertion(+), 2 deletions(-)

diff --git a/scripts/qapi/pylintrc b/scripts/qapi/pylintrc
index 76d54c30f8..507f15537a 100644
--- a/scripts/qapi/pylintrc
+++ b/scripts/qapi/pylintrc
@@ -2,8 +2,7 @@
 
 # Add files or directories matching the regex patterns to the ignore list.
 # The regex matches against base names, not paths.
-ignore-patterns=common.py,
-error.py,
+ignore-patterns=error.py,
 expr.py,
 gen.py,
 parser.py,
-- 
2.26.2

[PULL 21/34] qapi/commands.py: Don't re-bind to variable of different type

[Markus Armbruster]

From: John Snow 

Mypy isn't a fan of rebinding a variable with a new data type.
It's easy enough to avoid.

Signed-off-by: John Snow 
Reviewed-by: Eduardo Habkost 
Reviewed-by: Cleber Rosa 
Reviewed-by: Markus Armbruster 
Message-Id: <20201009161558.107041-22-js...@redhat.com>
Signed-off-by: Markus Armbruster 
---
 scripts/qapi/commands.py | 4 +---
 1 file changed, 1 insertion(+), 3 deletions(-)

diff --git a/scripts/qapi/commands.py b/scripts/qapi/commands.py
index 88ba11c40e..b0abb985a4 100644
--- a/scripts/qapi/commands.py
+++ b/scripts/qapi/commands.py
@@ -198,14 +198,12 @@ def gen_register_command(name, success_response, 
allow_oob, allow_preconfig,
 if not options:
 options = ['QCO_NO_OPTIONS']
 
-options = " | ".join(options)
-
 ret = mcgen('''
 qmp_register_command(cmds, "%(name)s",
  qmp_marshal_%(c_name)s, %(opts)s);
 ''',
 name=name, c_name=c_name(name),
-opts=options)
+opts=" | ".join(options))
 return ret
 
 
-- 
2.26.2

[PULL 08/34] qapi: delint using flake8

[Markus Armbruster]

From: John Snow 

Petty style guide fixes and line length enforcement.  Not a big win, not
a big loss, but flake8 passes 100% on the qapi module, which gives us an
easy baseline to enforce hereafter.

A note on the flake8 exception: flake8 will warn on *any* bare except,
but pylint's is context-aware and will suppress the warning if you
re-raise the exception.

Signed-off-by: John Snow 
Reviewed-by: Eduardo Habkost 
Reviewed-by: Cleber Rosa 
Message-Id: <20201009161558.107041-9-js...@redhat.com>
Reviewed-by: Markus Armbruster 
Signed-off-by: Markus Armbruster 
---
 scripts/qapi/.flake8 |  2 ++
 scripts/qapi/commands.py |  3 ++-
 scripts/qapi/schema.py   |  8 +---
 scripts/qapi/visit.py| 16 +++-
 4 files changed, 20 insertions(+), 9 deletions(-)
 create mode 100644 scripts/qapi/.flake8

diff --git a/scripts/qapi/.flake8 b/scripts/qapi/.flake8
new file mode 100644
index 00..6b158c68b8
--- /dev/null
+++ b/scripts/qapi/.flake8
@@ -0,0 +1,2 @@
+[flake8]
+extend-ignore = E722  # Prefer pylint's bare-except checks to flake8's
diff --git a/scripts/qapi/commands.py b/scripts/qapi/commands.py
index e06c10afcd..cde0c1e777 100644
--- a/scripts/qapi/commands.py
+++ b/scripts/qapi/commands.py
@@ -65,7 +65,8 @@ def gen_call(name, arg_type, boxed, ret_type):
 def gen_marshal_output(ret_type):
 return mcgen('''
 
-static void qmp_marshal_output_%(c_name)s(%(c_type)s ret_in, QObject 
**ret_out, Error **errp)
+static void qmp_marshal_output_%(c_name)s(%(c_type)s ret_in,
+QObject **ret_out, Error **errp)
 {
 Visitor *v;
 
diff --git a/scripts/qapi/schema.py b/scripts/qapi/schema.py
index 71ebb1e396..afd750989e 100644
--- a/scripts/qapi/schema.py
+++ b/scripts/qapi/schema.py
@@ -536,7 +536,7 @@ class QAPISchemaVariants:
 v.set_defined_in(name)
 
 def check(self, schema, seen):
-if not self.tag_member: # flat union
+if not self.tag_member:  # flat union
 self.tag_member = seen.get(c_name(self._tag_name))
 base = "'base'"
 # Pointing to the base type when not implicit would be
@@ -824,7 +824,7 @@ class QAPISchema:
 self._entity_dict = {}
 self._module_dict = OrderedDict()
 self._schema_dir = os.path.dirname(fname)
-self._make_module(None) # built-ins
+self._make_module(None)  # built-ins
 self._make_module(fname)
 self._predefining = True
 self._def_predefineds()
@@ -968,7 +968,9 @@ class QAPISchema:
 # But it's not tight: the disjunction need not imply it.  We
 # may end up compiling useless wrapper types.
 # TODO kill simple unions or implement the disjunction
-assert (ifcond or []) == typ._ifcond # pylint: 
disable=protected-access
+
+# pylint: disable=protected-access
+assert (ifcond or []) == typ._ifcond
 else:
 self._def_entity(QAPISchemaObjectType(
 name, info, None, ifcond, None, None, members, None))
diff --git a/scripts/qapi/visit.py b/scripts/qapi/visit.py
index ea277e7704..9fdbe5b9ef 100644
--- a/scripts/qapi/visit.py
+++ b/scripts/qapi/visit.py
@@ -31,7 +31,9 @@ def gen_visit_decl(name, scalar=False):
 if not scalar:
 c_type += '*'
 return mcgen('''
-bool visit_type_%(c_name)s(Visitor *v, const char *name, %(c_type)sobj, Error 
**errp);
+
+bool visit_type_%(c_name)s(Visitor *v, const char *name,
+ %(c_type)sobj, Error **errp);
 ''',
  c_name=c_name(name), c_type=c_type)
 
@@ -125,7 +127,8 @@ bool visit_type_%(c_name)s_members(Visitor *v, %(c_name)s 
*obj, Error **errp)
 def gen_visit_list(name, element_type):
 return mcgen('''
 
-bool visit_type_%(c_name)s(Visitor *v, const char *name, %(c_name)s **obj, 
Error **errp)
+bool visit_type_%(c_name)s(Visitor *v, const char *name,
+ %(c_name)s **obj, Error **errp)
 {
 bool ok = false;
 %(c_name)s *tail;
@@ -158,7 +161,8 @@ out_obj:
 def gen_visit_enum(name):
 return mcgen('''
 
-bool visit_type_%(c_name)s(Visitor *v, const char *name, %(c_name)s *obj, 
Error **errp)
+bool visit_type_%(c_name)s(Visitor *v, const char *name,
+ %(c_name)s *obj, Error **errp)
 {
 int value = *obj;
 bool ok = visit_type_enum(v, name, &value, &%(c_name)s_lookup, errp);
@@ -172,7 +176,8 @@ bool visit_type_%(c_name)s(Visitor *v, const char *name, 
%(c_name)s *obj, Error
 def gen_visit_alternate(name, variants):
 ret = mcgen('''
 
-bool visit_type_%(c_name)s(Visitor *v, const char *name, %(c_name)s **obj, 
Error **errp)
+bool visit_type_%(c_name)s(Visitor *v, const char *name,
+ %(c_name)s **obj, Error **errp)
 {
 bool ok = false;
 
@@ -247,7 +252,8 @@ out_obj:
 def gen_visit_object(name, base, members, variants):
 return mcgen('''
 
-bool visit_type_%(c_name)s(Visitor *v, const char *name, %(c_name)s **obj, 
Error **errp)
+bool visit_type_%(c_name)s(Visitor *v,

[PULL 18/34] qapi: establish mypy type-checking baseline

[Markus Armbruster]

From: John Snow 

Fix a minor typing issue, and then establish a mypy type-checking
baseline.

Like pylint, this should be run from the folder above:

 > mypy --config-file=qapi/mypy.ini qapi/

This is designed and tested for mypy 0.770 or greater.

Signed-off-by: John Snow 
Tested-by: Eduardo Habkost 
Reviewed-by: Cleber Rosa 
Tested-by: Cleber Rosa 
Message-Id: <20201009161558.107041-19-js...@redhat.com>
Reviewed-by: Markus Armbruster 
Signed-off-by: Markus Armbruster 
---
 scripts/qapi/mypy.ini  | 60 ++
 scripts/qapi/schema.py |  3 ++-
 2 files changed, 62 insertions(+), 1 deletion(-)
 create mode 100644 scripts/qapi/mypy.ini

diff --git a/scripts/qapi/mypy.ini b/scripts/qapi/mypy.ini
new file mode 100644
index 00..00fac15dc6
--- /dev/null
+++ b/scripts/qapi/mypy.ini
@@ -0,0 +1,60 @@
+[mypy]
+strict = True
+strict_optional = False
+disallow_untyped_calls = False
+python_version = 3.6
+
+[mypy-qapi.commands]
+disallow_untyped_defs = False
+disallow_incomplete_defs = False
+check_untyped_defs = False
+
+[mypy-qapi.error]
+disallow_untyped_defs = False
+disallow_incomplete_defs = False
+check_untyped_defs = False
+
+[mypy-qapi.events]
+disallow_untyped_defs = False
+disallow_incomplete_defs = False
+check_untyped_defs = False
+
+[mypy-qapi.expr]
+disallow_untyped_defs = False
+disallow_incomplete_defs = False
+check_untyped_defs = False
+
+[mypy-qapi.gen]
+disallow_untyped_defs = False
+disallow_incomplete_defs = False
+check_untyped_defs = False
+
+[mypy-qapi.introspect]
+disallow_untyped_defs = False
+disallow_incomplete_defs = False
+check_untyped_defs = False
+
+[mypy-qapi.parser]
+disallow_untyped_defs = False
+disallow_incomplete_defs = False
+check_untyped_defs = False
+
+[mypy-qapi.schema]
+disallow_untyped_defs = False
+disallow_incomplete_defs = False
+check_untyped_defs = False
+
+[mypy-qapi.source]
+disallow_untyped_defs = False
+disallow_incomplete_defs = False
+check_untyped_defs = False
+
+[mypy-qapi.types]
+disallow_untyped_defs = False
+disallow_incomplete_defs = False
+check_untyped_defs = False
+
+[mypy-qapi.visit]
+disallow_untyped_defs = False
+disallow_incomplete_defs = False
+check_untyped_defs = False
diff --git a/scripts/qapi/schema.py b/scripts/qapi/schema.py
index 7c01592956..720449feee 100644
--- a/scripts/qapi/schema.py
+++ b/scripts/qapi/schema.py
@@ -17,6 +17,7 @@
 from collections import OrderedDict
 import os
 import re
+from typing import Optional
 
 from .common import POINTER_SUFFIX, c_name
 from .error import QAPIError, QAPISemError
@@ -25,7 +26,7 @@ from .parser import QAPISchemaParser
 
 
 class QAPISchemaEntity:
-meta = None
+meta: Optional[str] = None
 
 def __init__(self, name, info, doc, ifcond=None, features=None):
 assert name is None or isinstance(name, str)
-- 
2.26.2

[PULL 15/34] qapi/common.py: add type hint annotations

[Markus Armbruster]

From: John Snow 

Annotations do not change runtime behavior.
This commit *only* adds annotations.

Note that build_params() cannot be fully annotated due to import
dependency issues.  The commit after next will take care of it.

Signed-off-by: John Snow 
Reviewed-by: Eduardo Habkost 
Reviewed-by: Cleber Rosa 
Message-Id: <20201009161558.107041-16-js...@redhat.com>
Reviewed-by: Markus Armbruster 
Signed-off-by: Markus Armbruster 
---
 scripts/qapi/common.py | 27 ---
 1 file changed, 16 insertions(+), 11 deletions(-)

diff --git a/scripts/qapi/common.py b/scripts/qapi/common.py
index 338adedef4..74a2c001ed 100644
--- a/scripts/qapi/common.py
+++ b/scripts/qapi/common.py
@@ -12,6 +12,7 @@
 # See the COPYING file in the top-level directory.
 
 import re
+from typing import Optional, Sequence
 
 
 EATSPACE = '\033EATSPACE.'
@@ -22,7 +23,7 @@ _C_NAME_TRANS = str.maketrans('.-', '__')
 # ENUMName -> ENUM_NAME, EnumName1 -> ENUM_NAME1
 # ENUM_NAME -> ENUM_NAME, ENUM_NAME1 -> ENUM_NAME1, ENUM_Name2 -> ENUM_NAME2
 # ENUM24_Name -> ENUM24_NAME
-def camel_to_upper(value):
+def camel_to_upper(value: str) -> str:
 c_fun_str = c_name(value, False)
 if value.isupper():
 return c_fun_str
@@ -41,7 +42,9 @@ def camel_to_upper(value):
 return new_name.lstrip('_').upper()
 
 
-def c_enum_const(type_name, const_name, prefix=None):
+def c_enum_const(type_name: str,
+ const_name: str,
+ prefix: Optional[str] = None) -> str:
 if prefix is not None:
 type_name = prefix
 return camel_to_upper(type_name) + '_' + c_name(const_name, False).upper()
@@ -56,7 +59,7 @@ def c_enum_const(type_name, const_name, prefix=None):
 # into substrings of a generated C function name.
 # '__a.b_c' -> '__a_b_c', 'x-foo' -> 'x_foo'
 # protect=True: 'int' -> 'q_int'; protect=False: 'int' -> 'int'
-def c_name(name, protect=True):
+def c_name(name: str, protect: bool = True) -> str:
 # ANSI X3J11/88-090, 3.1.1
 c89_words = set(['auto', 'break', 'case', 'char', 'const', 'continue',
  'default', 'do', 'double', 'else', 'enum', 'extern',
@@ -131,24 +134,24 @@ indent = Indentation()
 
 # Generate @code with @kwds interpolated.
 # Obey indent, and strip EATSPACE.
-def cgen(code, **kwds):
+def cgen(code: str, **kwds: object) -> str:
 raw = code % kwds
 if indent:
 raw = re.sub(r'^(?!(#|$))', str(indent), raw, flags=re.MULTILINE)
 return re.sub(re.escape(EATSPACE) + r' *', '', raw)
 
 
-def mcgen(code, **kwds):
+def mcgen(code: str, **kwds: object) -> str:
 if code[0] == '\n':
 code = code[1:]
 return cgen(code, **kwds)
 
 
-def c_fname(filename):
+def c_fname(filename: str) -> str:
 return re.sub(r'[^A-Za-z0-9_]', '_', filename)
 
 
-def guardstart(name):
+def guardstart(name: str) -> str:
 return mcgen('''
 #ifndef %(name)s
 #define %(name)s
@@ -157,7 +160,7 @@ def guardstart(name):
  name=c_fname(name).upper())
 
 
-def guardend(name):
+def guardend(name: str) -> str:
 return mcgen('''
 
 #endif /* %(name)s */
@@ -165,7 +168,7 @@ def guardend(name):
  name=c_fname(name).upper())
 
 
-def gen_if(ifcond):
+def gen_if(ifcond: Sequence[str]) -> str:
 ret = ''
 for ifc in ifcond:
 ret += mcgen('''
@@ -174,7 +177,7 @@ def gen_if(ifcond):
 return ret
 
 
-def gen_endif(ifcond):
+def gen_endif(ifcond: Sequence[str]) -> str:
 ret = ''
 for ifc in reversed(ifcond):
 ret += mcgen('''
@@ -183,7 +186,9 @@ def gen_endif(ifcond):
 return ret
 
 
-def build_params(arg_type, boxed, extra=None):
+def build_params(arg_type,
+ boxed: bool,
+ extra: Optional[str] = None) -> str:
 ret = ''
 sep = ''
 if boxed:
-- 
2.26.2

[PULL 27/34] qapi/gen.py: Remove unused parameter

[Markus Armbruster]

From: John Snow 

_module_dirname doesn't use the 'what' argument, so remove it.

Signed-off-by: John Snow 
Reviewed-by: Eduardo Habkost 
Reviewed-by: Cleber Rosa 
Message-Id: <20201009161558.107041-30-js...@redhat.com>
Reviewed-by: Markus Armbruster 
Signed-off-by: Markus Armbruster 
---
 scripts/qapi/gen.py | 4 ++--
 1 file changed, 2 insertions(+), 2 deletions(-)

diff --git a/scripts/qapi/gen.py b/scripts/qapi/gen.py
index 6b1007a035..8b851c6262 100644
--- a/scripts/qapi/gen.py
+++ b/scripts/qapi/gen.py
@@ -257,7 +257,7 @@ class QAPISchemaModularCVisitor(QAPISchemaVisitor):
 def _is_builtin_module(name: Optional[str]) -> bool:
 return not name
 
-def _module_dirname(self, what: str, name: Optional[str]) -> str:
+def _module_dirname(self, name: Optional[str]) -> str:
 if self._is_user_module(name):
 return os.path.dirname(name)
 return ''
@@ -275,7 +275,7 @@ class QAPISchemaModularCVisitor(QAPISchemaVisitor):
 return ret
 
 def _module_filename(self, what: str, name: Optional[str]) -> str:
-return os.path.join(self._module_dirname(what, name),
+return os.path.join(self._module_dirname(name),
 self._module_basename(what, name))
 
 def _add_module(self, name: Optional[str], blurb: str) -> None:
-- 
2.26.2

[PULL 17/34] qapi/common.py: move build_params into gen.py

[Markus Armbruster]

From: John Snow 

Including it in common.py creates a circular import dependency; schema
relies on common, but common.build_params requires a type annotation
from schema. To type this properly, it needs to be moved outside the
cycle.

Signed-off-by: John Snow 
Reviewed-by: Eduardo Habkost 
Reviewed-by: Cleber Rosa 
Message-Id: <20201009161558.107041-18-js...@redhat.com>
Reviewed-by: Markus Armbruster 
Signed-off-by: Markus Armbruster 
---
 scripts/qapi/commands.py |  9 +++--
 scripts/qapi/common.py   | 23 ---
 scripts/qapi/events.py   |  9 ++---
 scripts/qapi/gen.py  | 29 +++--
 4 files changed, 36 insertions(+), 34 deletions(-)

diff --git a/scripts/qapi/commands.py b/scripts/qapi/commands.py
index cde0c1e777..88ba11c40e 100644
--- a/scripts/qapi/commands.py
+++ b/scripts/qapi/commands.py
@@ -13,8 +13,13 @@ This work is licensed under the terms of the GNU GPL, 
version 2.
 See the COPYING file in the top-level directory.
 """
 
-from .common import build_params, c_name, mcgen
-from .gen import QAPIGenCCode, QAPISchemaModularCVisitor, ifcontext
+from .common import c_name, mcgen
+from .gen import (
+QAPIGenCCode,
+QAPISchemaModularCVisitor,
+build_params,
+ifcontext,
+)
 
 
 def gen_command_decl(name, arg_type, boxed, ret_type):
diff --git a/scripts/qapi/common.py b/scripts/qapi/common.py
index 669e3829b3..11b86beeab 100644
--- a/scripts/qapi/common.py
+++ b/scripts/qapi/common.py
@@ -210,26 +210,3 @@ def gen_endif(ifcond: Sequence[str]) -> str:
 #endif /* %(cond)s */
 ''', cond=ifc)
 return ret
-
-
-def build_params(arg_type,
- boxed: bool,
- extra: Optional[str] = None) -> str:
-ret = ''
-sep = ''
-if boxed:
-assert arg_type
-ret += '%s arg' % arg_type.c_param_type()
-sep = ', '
-elif arg_type:
-assert not arg_type.variants
-for memb in arg_type.members:
-ret += sep
-sep = ', '
-if memb.optional:
-ret += 'bool has_%s, ' % c_name(memb.name)
-ret += '%s %s' % (memb.type.c_param_type(),
-  c_name(memb.name))
-if extra:
-ret += sep + extra
-return ret if ret else 'void'
diff --git a/scripts/qapi/events.py b/scripts/qapi/events.py
index 6b3afa14d7..f840a62ed9 100644
--- a/scripts/qapi/events.py
+++ b/scripts/qapi/events.py
@@ -12,13 +12,8 @@ This work is licensed under the terms of the GNU GPL, 
version 2.
 See the COPYING file in the top-level directory.
 """
 
-from .common import (
-build_params,
-c_enum_const,
-c_name,
-mcgen,
-)
-from .gen import QAPISchemaModularCVisitor, ifcontext
+from .common import c_enum_const, c_name, mcgen
+from .gen import QAPISchemaModularCVisitor, build_params, ifcontext
 from .schema import QAPISchemaEnumMember
 from .types import gen_enum, gen_enum_lookup
 
diff --git a/scripts/qapi/gen.py b/scripts/qapi/gen.py
index 1fed712b43..fff0c0acb6 100644
--- a/scripts/qapi/gen.py
+++ b/scripts/qapi/gen.py
@@ -2,7 +2,7 @@
 #
 # QAPI code generation
 #
-# Copyright (c) 2018-2019 Red Hat Inc.
+# Copyright (c) 2015-2019 Red Hat Inc.
 #
 # Authors:
 #  Markus Armbruster 
@@ -15,16 +15,18 @@ from contextlib import contextmanager
 import errno
 import os
 import re
+from typing import Optional
 
 from .common import (
 c_fname,
+c_name,
 gen_endif,
 gen_if,
 guardend,
 guardstart,
 mcgen,
 )
-from .schema import QAPISchemaVisitor
+from .schema import QAPISchemaObjectType, QAPISchemaVisitor
 
 
 class QAPIGen:
@@ -90,6 +92,29 @@ def _wrap_ifcond(ifcond, before, after):
 return out
 
 
+def build_params(arg_type: Optional[QAPISchemaObjectType],
+ boxed: bool,
+ extra: Optional[str] = None) -> str:
+ret = ''
+sep = ''
+if boxed:
+assert arg_type
+ret += '%s arg' % arg_type.c_param_type()
+sep = ', '
+elif arg_type:
+assert not arg_type.variants
+for memb in arg_type.members:
+ret += sep
+sep = ', '
+if memb.optional:
+ret += 'bool has_%s, ' % c_name(memb.name)
+ret += '%s %s' % (memb.type.c_param_type(),
+  c_name(memb.name))
+if extra:
+ret += sep + extra
+return ret if ret else 'void'
+
+
 class QAPIGenCCode(QAPIGen):
 
 def __init__(self, fname):
-- 
2.26.2

[PULL 23/34] qapi/source.py: add type hint annotations

[Markus Armbruster]

From: John Snow 

Annotations do not change runtime behavior.
This commit *only* adds annotations.

A note on typing of __init__: mypy requires init functions with no
parameters to document a return type of None to be considered fully
typed. In the case when there are input parameters, None may be omitted.

Since __init__ may never return any value, it is preferred to omit the
return annotation whenever possible.

Signed-off-by: John Snow 
Reviewed-by: Eduardo Habkost 
Reviewed-by: Cleber Rosa 
Tested-by: Cleber Rosa 
Message-Id: <20201009161558.107041-25-js...@redhat.com>
Reviewed-by: Markus Armbruster 
Signed-off-by: Markus Armbruster 
---
 scripts/qapi/mypy.ini  |  5 -
 scripts/qapi/source.py | 32 +++-
 2 files changed, 19 insertions(+), 18 deletions(-)

diff --git a/scripts/qapi/mypy.ini b/scripts/qapi/mypy.ini
index 8ab9ac52cc..1b8555dfa3 100644
--- a/scripts/qapi/mypy.ini
+++ b/scripts/qapi/mypy.ini
@@ -34,11 +34,6 @@ disallow_untyped_defs = False
 disallow_incomplete_defs = False
 check_untyped_defs = False
 
-[mypy-qapi.source]
-disallow_untyped_defs = False
-disallow_incomplete_defs = False
-check_untyped_defs = False
-
 [mypy-qapi.types]
 disallow_untyped_defs = False
 disallow_incomplete_defs = False
diff --git a/scripts/qapi/source.py b/scripts/qapi/source.py
index e97b9a8e15..27af5295a8 100644
--- a/scripts/qapi/source.py
+++ b/scripts/qapi/source.py
@@ -11,37 +11,43 @@
 
 import copy
 import sys
+from typing import List, Optional, TypeVar
 
 
 class QAPISchemaPragma:
-def __init__(self):
+def __init__(self) -> None:
 # Are documentation comments required?
 self.doc_required = False
 # Whitelist of commands allowed to return a non-dictionary
-self.returns_whitelist = []
+self.returns_whitelist: List[str] = []
 # Whitelist of entities allowed to violate case conventions
-self.name_case_whitelist = []
+self.name_case_whitelist: List[str] = []
 
 
 class QAPISourceInfo:
-def __init__(self, fname, line, parent):
+T = TypeVar('T', bound='QAPISourceInfo')
+
+def __init__(self, fname: str, line: int,
+ parent: Optional['QAPISourceInfo']):
 self.fname = fname
 self.line = line
 self.parent = parent
-self.pragma = parent.pragma if parent else QAPISchemaPragma()
-self.defn_meta = None
-self.defn_name = None
+self.pragma: QAPISchemaPragma = (
+parent.pragma if parent else QAPISchemaPragma()
+)
+self.defn_meta: Optional[str] = None
+self.defn_name: Optional[str] = None
 
-def set_defn(self, meta, name):
+def set_defn(self, meta: str, name: str) -> None:
 self.defn_meta = meta
 self.defn_name = name
 
-def next_line(self):
+def next_line(self: T) -> T:
 info = copy.copy(self)
 info.line += 1
 return info
 
-def loc(self):
+def loc(self) -> str:
 if self.fname is None:
 return sys.argv[0]
 ret = self.fname
@@ -49,13 +55,13 @@ class QAPISourceInfo:
 ret += ':%d' % self.line
 return ret
 
-def in_defn(self):
+def in_defn(self) -> str:
 if self.defn_name:
 return "%s: In %s '%s':\n" % (self.fname,
   self.defn_meta, self.defn_name)
 return ''
 
-def include_path(self):
+def include_path(self) -> str:
 ret = ''
 parent = self.parent
 while parent:
@@ -63,5 +69,5 @@ class QAPISourceInfo:
 parent = parent.parent
 return ret
 
-def __str__(self):
+def __str__(self) -> str:
 return self.include_path() + self.in_defn() + self.loc()
-- 
2.26.2

[PULL 20/34] qapi/events.py: Move comments into docstrings

[Markus Armbruster]

From: John Snow 

Clarify them while we're here.

Signed-off-by: John Snow 
Reviewed-by: Eduardo Habkost 
Reviewed-by: Cleber Rosa 
Message-Id: <20201009161558.107041-21-js...@redhat.com>
Reviewed-by: Markus Armbruster 
Signed-off-by: Markus Armbruster 
---
 scripts/qapi/events.py | 6 +-
 1 file changed, 5 insertions(+), 1 deletion(-)

diff --git a/scripts/qapi/events.py b/scripts/qapi/events.py
index 57e0939e96..599f3d1f56 100644
--- a/scripts/qapi/events.py
+++ b/scripts/qapi/events.py
@@ -44,8 +44,12 @@ def gen_event_send_decl(name: str,
  proto=build_event_send_proto(name, arg_type, boxed))
 
 
-# Declare and initialize an object 'qapi' using parameters from build_params()
 def gen_param_var(typ: QAPISchemaObjectType) -> str:
+"""
+Generate a struct variable holding the event parameters.
+
+Initialize it with the function arguments defined in `gen_event_send`.
+"""
 assert not typ.variants
 ret = mcgen('''
 %(c_name)s param = {
-- 
2.26.2

[PULL 29/34] qapi/gen.py: delint with pylint

[Markus Armbruster]

From: John Snow 

'fp' and 'fd' are self-evident in context, add them to the list of OK
names.

_top and _bottom also need to stay standard methods because some users
override the method and need to use `self`. Tell pylint to shush.

Signed-off-by: John Snow 
Reviewed-by: Eduardo Habkost 
Reviewed-by: Cleber Rosa 
Message-Id: <20201009161558.107041-32-js...@redhat.com>
Reviewed-by: Markus Armbruster 
Signed-off-by: Markus Armbruster 
---
 scripts/qapi/gen.py   | 2 ++
 scripts/qapi/pylintrc | 5 +++--
 2 files changed, 5 insertions(+), 2 deletions(-)

diff --git a/scripts/qapi/gen.py b/scripts/qapi/gen.py
index 670c21e210..b40f18eee3 100644
--- a/scripts/qapi/gen.py
+++ b/scripts/qapi/gen.py
@@ -51,9 +51,11 @@ class QAPIGen:
 return self._top() + self._preamble + self._body + self._bottom()
 
 def _top(self) -> str:
+# pylint: disable=no-self-use
 return ''
 
 def _bottom(self) -> str:
+# pylint: disable=no-self-use
 return ''
 
 def write(self, output_dir: str) -> None:
diff --git a/scripts/qapi/pylintrc b/scripts/qapi/pylintrc
index d840b15031..8badcb11cd 100644
--- a/scripts/qapi/pylintrc
+++ b/scripts/qapi/pylintrc
@@ -4,7 +4,6 @@
 # The regex matches against base names, not paths.
 ignore-patterns=error.py,
 expr.py,
-gen.py,
 parser.py,
 schema.py,
 types.py,
@@ -45,7 +44,9 @@ good-names=i,
k,
ex,
Run,
-   _
+   _,
+   fp,  # fp = open(...)
+   fd,  # fd = os.open(...)
 
 [VARIABLES]
 
-- 
2.26.2

[PULL 02/34] qapi: modify docstrings to be sphinx-compatible

[Markus Armbruster]

From: John Snow 

A precise style guide and a package-wide overhaul is forthcoming pending
further discussion and consensus. For now, merely avoid obvious errors
that cause Sphinx documentation build problems, using a style loosely
based on PEP 257 and Sphinx Autodoc. It is chosen for interoperability
with our existing Sphinx framework, and because it has loose recognition
in the Pycharm IDE.

See also:

https://www.python.org/dev/peps/pep-0257/
https://www.sphinx-doc.org/en/master/usage/restructuredtext/domains.html#info-field-lists
Signed-off-by: John Snow 
Message-Id: <20201009161558.107041-3-js...@redhat.com>
Reviewed-by: Markus Armbruster 
Signed-off-by: Markus Armbruster 
---
 scripts/qapi/gen.py | 6 --
 1 file changed, 4 insertions(+), 2 deletions(-)

diff --git a/scripts/qapi/gen.py b/scripts/qapi/gen.py
index ca66c82b5b..dc7b94aa11 100644
--- a/scripts/qapi/gen.py
+++ b/scripts/qapi/gen.py
@@ -154,9 +154,11 @@ class QAPIGenH(QAPIGenC):
 
 @contextmanager
 def ifcontext(ifcond, *args):
-"""A 'with' statement context manager to wrap with start_if()/end_if()
+"""
+A with-statement context manager that wraps with `start_if()` / `end_if()`.
 
-*args: any number of QAPIGenCCode
+:param ifcond: A list of conditionals, passed to `start_if()`.
+:param args: any number of `QAPIGenCCode`.
 
 Example::
 
-- 
2.26.2

[PULL 16/34] qapi/common.py: Convert comments into docstrings, and elaborate

[Markus Armbruster]

From: John Snow 

As docstrings, they'll show up in documentation and IDE help.

The docstring style being targeted is the Sphinx documentation
style. Sphinx uses an extension of ReST with "domains". We use the
(implicit) Python domain, which supports a number of custom "info
fields". Those info fields are documented here:
https://www.sphinx-doc.org/en/master/usage/restructuredtext/domains.html#info-field-lists

Primarily, we use `:param X: descr`, `:return[s]: descr`, and `:raise[s]
Z: when`. Everything else is the Sphinx dialect of ReST.

(No, nothing checks or enforces this style that I am aware of. Sphinx
either chokes or succeeds, but does not enforce a standard of what is
otherwise inside the docstring. Pycharm does highlight when your param
fields are not aligned with the actual fields present. It does not
highlight missing return or exception statements. There is no existing
style guide I am aware of that covers a standard for a minimally
acceptable docstring. I am debating writing one.)

Signed-off-by: John Snow 
Reviewed-by: Eduardo Habkost 
Reviewed-by: Cleber Rosa 
Message-Id: <20201009161558.107041-17-js...@redhat.com>
Reviewed-by: Markus Armbruster 
Signed-off-by: Markus Armbruster 
---
 scripts/qapi/common.py | 54 +++---
 1 file changed, 40 insertions(+), 14 deletions(-)

diff --git a/scripts/qapi/common.py b/scripts/qapi/common.py
index 74a2c001ed..669e3829b3 100644
--- a/scripts/qapi/common.py
+++ b/scripts/qapi/common.py
@@ -15,15 +15,25 @@ import re
 from typing import Optional, Sequence
 
 
+#: Magic string that gets removed along with all space to its right.
 EATSPACE = '\033EATSPACE.'
 POINTER_SUFFIX = ' *' + EATSPACE
 _C_NAME_TRANS = str.maketrans('.-', '__')
 
 
-# ENUMName -> ENUM_NAME, EnumName1 -> ENUM_NAME1
-# ENUM_NAME -> ENUM_NAME, ENUM_NAME1 -> ENUM_NAME1, ENUM_Name2 -> ENUM_NAME2
-# ENUM24_Name -> ENUM24_NAME
 def camel_to_upper(value: str) -> str:
+"""
+Converts CamelCase to CAMEL_CASE.
+
+Examples::
+
+ENUMName -> ENUM_NAME
+EnumName1 -> ENUM_NAME1
+ENUM_NAME -> ENUM_NAME
+ENUM_NAME1 -> ENUM_NAME1
+ENUM_Name2 -> ENUM_NAME2
+ENUM24_Name -> ENUM24_NAME
+"""
 c_fun_str = c_name(value, False)
 if value.isupper():
 return c_fun_str
@@ -45,21 +55,33 @@ def camel_to_upper(value: str) -> str:
 def c_enum_const(type_name: str,
  const_name: str,
  prefix: Optional[str] = None) -> str:
+"""
+Generate a C enumeration constant name.
+
+:param type_name: The name of the enumeration.
+:param const_name: The name of this constant.
+:param prefix: Optional, prefix that overrides the type_name.
+"""
 if prefix is not None:
 type_name = prefix
 return camel_to_upper(type_name) + '_' + c_name(const_name, False).upper()
 
 
-# Map @name to a valid C identifier.
-# If @protect, avoid returning certain ticklish identifiers (like
-# C keywords) by prepending 'q_'.
-#
-# Used for converting 'name' from a 'name':'type' qapi definition
-# into a generated struct member, as well as converting type names
-# into substrings of a generated C function name.
-# '__a.b_c' -> '__a_b_c', 'x-foo' -> 'x_foo'
-# protect=True: 'int' -> 'q_int'; protect=False: 'int' -> 'int'
 def c_name(name: str, protect: bool = True) -> str:
+"""
+Map ``name`` to a valid C identifier.
+
+Used for converting 'name' from a 'name':'type' qapi definition
+into a generated struct member, as well as converting type names
+into substrings of a generated C function name.
+
+'__a.b_c' -> '__a_b_c', 'x-foo' -> 'x_foo'
+protect=True: 'int' -> 'q_int'; protect=False: 'int' -> 'int'
+
+:param name: The name to map.
+:param protect: If true, avoid returning certain ticklish identifiers
+(like C keywords) by prepending ``q_``.
+"""
 # ANSI X3J11/88-090, 3.1.1
 c89_words = set(['auto', 'break', 'case', 'char', 'const', 'continue',
  'default', 'do', 'double', 'else', 'enum', 'extern',
@@ -129,12 +151,16 @@ class Indentation:
 self._level -= amount
 
 
+#: Global, current indent level for code generation.
 indent = Indentation()
 
 
-# Generate @code with @kwds interpolated.
-# Obey indent, and strip EATSPACE.
 def cgen(code: str, **kwds: object) -> str:
+"""
+Generate ``code`` with ``kwds`` interpolated.
+
+Obey `indent`, and strip `EATSPACE`.
+"""
 raw = code % kwds
 if indent:
 raw = re.sub(r'^(?!(#|$))', str(indent), raw, flags=re.MULTILINE)
-- 
2.26.2

[PULL 24/34] qapi/source.py: delint with pylint

[Markus Armbruster]

From: John Snow 

Shush an error and leave a hint for future cleanups when we're allowed
to use Python 3.7+.

Signed-off-by: John Snow 
Reviewed-by: Eduardo Habkost 
Reviewed-by: Cleber Rosa 
Tested-by: Cleber Rosa 
Message-Id: <20201009161558.107041-26-js...@redhat.com>
Reviewed-by: Markus Armbruster 
Signed-off-by: Markus Armbruster 
---
 scripts/qapi/pylintrc  | 1 -
 scripts/qapi/source.py | 3 +++
 2 files changed, 3 insertions(+), 1 deletion(-)

diff --git a/scripts/qapi/pylintrc b/scripts/qapi/pylintrc
index 507f15537a..d840b15031 100644
--- a/scripts/qapi/pylintrc
+++ b/scripts/qapi/pylintrc
@@ -7,7 +7,6 @@ ignore-patterns=error.py,
 gen.py,
 parser.py,
 schema.py,
-source.py,
 types.py,
 visit.py,
 
diff --git a/scripts/qapi/source.py b/scripts/qapi/source.py
index 27af5295a8..d7a79a9b8a 100644
--- a/scripts/qapi/source.py
+++ b/scripts/qapi/source.py
@@ -15,6 +15,9 @@ from typing import List, Optional, TypeVar
 
 
 class QAPISchemaPragma:
+# Replace with @dataclass in Python 3.7+
+# pylint: disable=too-few-public-methods
+
 def __init__(self) -> None:
 # Are documentation comments required?
 self.doc_required = False
-- 
2.26.2

[PULL 34/34] qapi/visit.py: add type hint annotations

[Markus Armbruster]

From: John Snow 

Annotations do not change runtime behavior.
This commit *only* adds annotations.

Signed-off-by: John Snow 
Reviewed-by: Eduardo Habkost 
Reviewed-by: Cleber Rosa 
Tested-by: Cleber Rosa 
Message-Id: <20201009161558.107041-37-js...@redhat.com>
Reviewed-by: Markus Armbruster 
Signed-off-by: Markus Armbruster 
---
 scripts/qapi/mypy.ini |  5 ---
 scripts/qapi/visit.py | 73 +--
 2 files changed, 56 insertions(+), 22 deletions(-)

diff --git a/scripts/qapi/mypy.ini b/scripts/qapi/mypy.ini
index 83f1981355..74fc6c8215 100644
--- a/scripts/qapi/mypy.ini
+++ b/scripts/qapi/mypy.ini
@@ -28,8 +28,3 @@ check_untyped_defs = False
 disallow_untyped_defs = False
 disallow_incomplete_defs = False
 check_untyped_defs = False
-
-[mypy-qapi.visit]
-disallow_untyped_defs = False
-disallow_incomplete_defs = False
-check_untyped_defs = False
diff --git a/scripts/qapi/visit.py b/scripts/qapi/visit.py
index 8699e5c09c..339f152152 100644
--- a/scripts/qapi/visit.py
+++ b/scripts/qapi/visit.py
@@ -13,6 +13,8 @@ This work is licensed under the terms of the GNU GPL, version 
2.
 See the COPYING file in the top-level directory.
 """
 
+from typing import List, Optional
+
 from .common import (
 c_enum_const,
 c_name,
@@ -22,10 +24,20 @@ from .common import (
 mcgen,
 )
 from .gen import QAPISchemaModularCVisitor, ifcontext
-from .schema import QAPISchemaEnumType, QAPISchemaObjectType
+from .schema import (
+QAPISchema,
+QAPISchemaEnumMember,
+QAPISchemaEnumType,
+QAPISchemaFeature,
+QAPISchemaObjectType,
+QAPISchemaObjectTypeMember,
+QAPISchemaType,
+QAPISchemaVariants,
+)
+from .source import QAPISourceInfo
 
 
-def gen_visit_decl(name, scalar=False):
+def gen_visit_decl(name: str, scalar: bool = False) -> str:
 c_type = c_name(name) + ' *'
 if not scalar:
 c_type += '*'
@@ -37,7 +49,7 @@ bool visit_type_%(c_name)s(Visitor *v, const char *name,
  c_name=c_name(name), c_type=c_type)
 
 
-def gen_visit_members_decl(name):
+def gen_visit_members_decl(name: str) -> str:
 return mcgen('''
 
 bool visit_type_%(c_name)s_members(Visitor *v, %(c_name)s *obj, Error **errp);
@@ -45,7 +57,10 @@ bool visit_type_%(c_name)s_members(Visitor *v, %(c_name)s 
*obj, Error **errp);
  c_name=c_name(name))
 
 
-def gen_visit_object_members(name, base, members, variants):
+def gen_visit_object_members(name: str,
+ base: Optional[QAPISchemaObjectType],
+ members: List[QAPISchemaObjectTypeMember],
+ variants: Optional[QAPISchemaVariants]) -> str:
 ret = mcgen('''
 
 bool visit_type_%(c_name)s_members(Visitor *v, %(c_name)s *obj, Error **errp)
@@ -125,7 +140,7 @@ bool visit_type_%(c_name)s_members(Visitor *v, %(c_name)s 
*obj, Error **errp)
 return ret
 
 
-def gen_visit_list(name, element_type):
+def gen_visit_list(name: str, element_type: QAPISchemaType) -> str:
 return mcgen('''
 
 bool visit_type_%(c_name)s(Visitor *v, const char *name,
@@ -159,7 +174,7 @@ out_obj:
  c_name=c_name(name), c_elt_type=element_type.c_name())
 
 
-def gen_visit_enum(name):
+def gen_visit_enum(name: str) -> str:
 return mcgen('''
 
 bool visit_type_%(c_name)s(Visitor *v, const char *name,
@@ -174,7 +189,7 @@ bool visit_type_%(c_name)s(Visitor *v, const char *name,
  c_name=c_name(name))
 
 
-def gen_visit_alternate(name, variants):
+def gen_visit_alternate(name: str, variants: QAPISchemaVariants) -> str:
 ret = mcgen('''
 
 bool visit_type_%(c_name)s(Visitor *v, const char *name,
@@ -250,7 +265,7 @@ out_obj:
 return ret
 
 
-def gen_visit_object(name):
+def gen_visit_object(name: str) -> str:
 return mcgen('''
 
 bool visit_type_%(c_name)s(Visitor *v, const char *name,
@@ -285,12 +300,12 @@ out_obj:
 
 class QAPISchemaGenVisitVisitor(QAPISchemaModularCVisitor):
 
-def __init__(self, prefix):
+def __init__(self, prefix: str):
 super().__init__(
 prefix, 'qapi-visit', ' * Schema-defined QAPI visitors',
 ' * Built-in QAPI visitors', __doc__)
 
-def _begin_system_module(self, name):
+def _begin_system_module(self, name: None) -> None:
 self._genc.preamble_add(mcgen('''
 #include "qemu/osdep.h"
 #include "qapi/error.h"
@@ -302,7 +317,7 @@ class QAPISchemaGenVisitVisitor(QAPISchemaModularCVisitor):
 
 '''))
 
-def _begin_user_module(self, name):
+def _begin_user_module(self, name: str) -> None:
 types = self._module_basename('qapi-types', name)
 visit = self._module_basename('qapi-visit', name)
 self._genc.preamble_add(mcgen('''
@@ -319,18 +334,34 @@ class 
QAPISchemaGenVisitVisitor(QAPISchemaModularCVisitor):
 ''',
   types=types))
 
-def visit_enum_type(self, name, info, ifcond, features, members, prefix):
+def visit_enum_type(self,
+n

[PULL 25/34] qapi/gen: Make _is_user_module() return bool

[Markus Armbruster]

From: John Snow 

_is_user_module() returns thruth values.  The next commit wants it to
return bool.  Make it so.

Signed-off-by: John Snow 
Reviewed-by: Cleber Rosa 
Reviewed-by: Eduardo Habkost 
Reviewed-by: Philippe Mathieu-Daudé 
Message-Id: <20201009161558.107041-27-js...@redhat.com>
Reviewed-by: Markus Armbruster 
[Commit message rewritten]
Signed-off-by: Markus Armbruster 
---
 scripts/qapi/gen.py | 2 +-
 1 file changed, 1 insertion(+), 1 deletion(-)

diff --git a/scripts/qapi/gen.py b/scripts/qapi/gen.py
index fff0c0acb6..2c305c4f82 100644
--- a/scripts/qapi/gen.py
+++ b/scripts/qapi/gen.py
@@ -241,7 +241,7 @@ class QAPISchemaModularCVisitor(QAPISchemaVisitor):
 
 @staticmethod
 def _is_user_module(name):
-return name and not name.startswith('./')
+return bool(name and not name.startswith('./'))
 
 @staticmethod
 def _is_builtin_module(name):
-- 
2.26.2

[PULL 26/34] qapi/gen.py: add type hint annotations

[Markus Armbruster]

From: John Snow 

Annotations do not change runtime behavior.
This commit *only* adds annotations.

Signed-off-by: John Snow 
Reviewed-by: Eduardo Habkost 
Reviewed-by: Cleber Rosa 
Message-Id: <20201009161558.107041-28-js...@redhat.com>
Message-Id: <20201009161558.107041-29-js...@redhat.com>
[mypy.ini update squashed in]
Reviewed-by: Markus Armbruster 
Signed-off-by: Markus Armbruster 
---
 scripts/qapi/gen.py   | 104 +++---
 scripts/qapi/mypy.ini |   5 --
 2 files changed, 57 insertions(+), 52 deletions(-)

diff --git a/scripts/qapi/gen.py b/scripts/qapi/gen.py
index 2c305c4f82..6b1007a035 100644
--- a/scripts/qapi/gen.py
+++ b/scripts/qapi/gen.py
@@ -15,7 +15,13 @@ from contextlib import contextmanager
 import errno
 import os
 import re
-from typing import Optional
+from typing import (
+Dict,
+Iterator,
+List,
+Optional,
+Tuple,
+)
 
 from .common import (
 c_fname,
@@ -27,31 +33,31 @@ from .common import (
 mcgen,
 )
 from .schema import QAPISchemaObjectType, QAPISchemaVisitor
+from .source import QAPISourceInfo
 
 
 class QAPIGen:
-
-def __init__(self, fname):
+def __init__(self, fname: Optional[str]):
 self.fname = fname
 self._preamble = ''
 self._body = ''
 
-def preamble_add(self, text):
+def preamble_add(self, text: str) -> None:
 self._preamble += text
 
-def add(self, text):
+def add(self, text: str) -> None:
 self._body += text
 
-def get_content(self):
+def get_content(self) -> str:
 return self._top() + self._preamble + self._body + self._bottom()
 
-def _top(self):
+def _top(self) -> str:
 return ''
 
-def _bottom(self):
+def _bottom(self) -> str:
 return ''
 
-def write(self, output_dir):
+def write(self, output_dir: str) -> None:
 # Include paths starting with ../ are used to reuse modules of the main
 # schema in specialised schemas. Don't overwrite the files that are
 # already generated for the main schema.
@@ -76,7 +82,7 @@ class QAPIGen:
 f.close()
 
 
-def _wrap_ifcond(ifcond, before, after):
+def _wrap_ifcond(ifcond: List[str], before: str, after: str) -> str:
 if before == after:
 return after   # suppress empty #if ... #endif
 
@@ -116,40 +122,38 @@ def build_params(arg_type: Optional[QAPISchemaObjectType],
 
 
 class QAPIGenCCode(QAPIGen):
-
-def __init__(self, fname):
+def __init__(self, fname: Optional[str]):
 super().__init__(fname)
-self._start_if = None
+self._start_if: Optional[Tuple[List[str], str, str]] = None
 
-def start_if(self, ifcond):
+def start_if(self, ifcond: List[str]) -> None:
 assert self._start_if is None
 self._start_if = (ifcond, self._body, self._preamble)
 
-def end_if(self):
+def end_if(self) -> None:
 assert self._start_if
 self._wrap_ifcond()
 self._start_if = None
 
-def _wrap_ifcond(self):
+def _wrap_ifcond(self) -> None:
 self._body = _wrap_ifcond(self._start_if[0],
   self._start_if[1], self._body)
 self._preamble = _wrap_ifcond(self._start_if[0],
   self._start_if[2], self._preamble)
 
-def get_content(self):
+def get_content(self) -> str:
 assert self._start_if is None
 return super().get_content()
 
 
 class QAPIGenC(QAPIGenCCode):
-
-def __init__(self, fname, blurb, pydoc):
+def __init__(self, fname: str, blurb: str, pydoc: str):
 super().__init__(fname)
 self._blurb = blurb
 self._copyright = '\n * '.join(re.findall(r'^Copyright .*', pydoc,
   re.MULTILINE))
 
-def _top(self):
+def _top(self) -> str:
 return mcgen('''
 /* AUTOMATICALLY GENERATED, DO NOT MODIFY */
 
@@ -165,7 +169,7 @@ class QAPIGenC(QAPIGenCCode):
 ''',
  blurb=self._blurb, copyright=self._copyright)
 
-def _bottom(self):
+def _bottom(self) -> str:
 return mcgen('''
 
 /* Dummy declaration to prevent empty .o file */
@@ -175,16 +179,15 @@ char qapi_dummy_%(name)s;
 
 
 class QAPIGenH(QAPIGenC):
-
-def _top(self):
+def _top(self) -> str:
 return super()._top() + guardstart(self.fname)
 
-def _bottom(self):
+def _bottom(self) -> str:
 return guardend(self.fname)
 
 
 @contextmanager
-def ifcontext(ifcond, *args):
+def ifcontext(ifcond: List[str], *args: QAPIGenCCode) -> Iterator[None]:
 """
 A with-statement context manager that wraps with `start_if()` / `end_if()`.
 
@@ -212,8 +215,11 @@ def ifcontext(ifcond, *args):
 
 
 class QAPISchemaMonolithicCVisitor(QAPISchemaVisitor):
-
-def __init__(self, prefix, what, blurb, pydoc):
+def __init__(self,
+ prefix: str,
+ what: str,
+ blurb: str,
+ pydoc: str):
 s

[PULL 09/34] qapi: add pylintrc

[Markus Armbruster]

From: John Snow 

Using `pylint --generate-rcfile > pylintrc`, generate a skeleton
pylintrc file. Sections that are not presently relevant (by the end of
this series) are removed leaving just the empty section as a search
engine / documentation hint to future authors.

I am targeting pylint 2.6.0. In the future (and hopefully before 5.2 is
released), I aim to have gitlab CI running the specific targeted
versions of pylint, mypy, flake8, etc in a job.

2.5.x will work if you additionally pass --disable=bad-whitespace.
This warning was removed from 2.6.x, for lack of consistent support.

Right now, quite a few modules are ignored as they are known to fail as
of this commit. modules will be removed from the known-bad list
throughout this and following series as they are repaired.

Note: Normally, pylintrc would go in the folder above the module, but as
that folder is shared by many things, it is going inside the module
folder (for now). Due to a bug in pylint 2.5+, pylint does not
correctly recognize when it is being run from "inside" a package, and
must be run *outside* of the package.

Therefore, to run it, you must:

 > pylint scripts/qapi/ --rcfile=scripts/qapi/pylintrc

Signed-off-by: John Snow 
Tested-by: Eduardo Habkost 
Reviewed-by: Cleber Rosa 
Tested-by: Cleber Rosa 
Message-Id: <20201009161558.107041-10-js...@redhat.com>
Reviewed-by: Markus Armbruster 
Signed-off-by: Markus Armbruster 
---
 scripts/qapi/pylintrc | 73 +++
 1 file changed, 73 insertions(+)
 create mode 100644 scripts/qapi/pylintrc

diff --git a/scripts/qapi/pylintrc b/scripts/qapi/pylintrc
new file mode 100644
index 00..76d54c30f8
--- /dev/null
+++ b/scripts/qapi/pylintrc
@@ -0,0 +1,73 @@
+[MASTER]
+
+# Add files or directories matching the regex patterns to the ignore list.
+# The regex matches against base names, not paths.
+ignore-patterns=common.py,
+error.py,
+expr.py,
+gen.py,
+parser.py,
+schema.py,
+source.py,
+types.py,
+visit.py,
+
+
+[MESSAGES CONTROL]
+
+# Disable the message, report, category or checker with the given id(s). You
+# can either give multiple identifiers separated by comma (,) or put this
+# option multiple times (only on the command line, not in the configuration
+# file where it should appear only once). You can also use "--disable=all" to
+# disable everything first and then reenable specific checks. For example, if
+# you want to run only the similarities checker, you can use "--disable=all
+# --enable=similarities". If you want to run only the classes checker, but have
+# no Warning level messages displayed, use "--disable=all --enable=classes
+# --disable=W".
+disable=fixme,
+missing-docstring,
+too-many-arguments,
+too-many-branches,
+too-many-statements,
+too-many-instance-attributes,
+
+[REPORTS]
+
+[REFACTORING]
+
+[MISCELLANEOUS]
+
+[LOGGING]
+
+[BASIC]
+
+# Good variable names which should always be accepted, separated by a comma.
+good-names=i,
+   j,
+   k,
+   ex,
+   Run,
+   _
+
+[VARIABLES]
+
+[STRING]
+
+[SPELLING]
+
+[FORMAT]
+
+[SIMILARITIES]
+
+# Ignore import statements themselves when computing similarities.
+ignore-imports=yes
+
+[TYPECHECK]
+
+[CLASSES]
+
+[IMPORTS]
+
+[DESIGN]
+
+[EXCEPTIONS]
-- 
2.26.2

[PULL 30/34] qapi/types.py: add type hint annotations

[Markus Armbruster]

From: John Snow 

Annotations do not change runtime behavior.
This commit *only* adds annotations.

Signed-off-by: John Snow 
Reviewed-by: Eduardo Habkost 
Reviewed-by: Cleber Rosa 
Message-Id: <20201009161558.107041-33-js...@redhat.com>
Reviewed-by: Markus Armbruster 
Signed-off-by: Markus Armbruster 
---
 scripts/qapi/mypy.ini |  5 ---
 scripts/qapi/types.py | 86 ---
 2 files changed, 64 insertions(+), 27 deletions(-)

diff --git a/scripts/qapi/mypy.ini b/scripts/qapi/mypy.ini
index c6960ff2db..83f1981355 100644
--- a/scripts/qapi/mypy.ini
+++ b/scripts/qapi/mypy.ini
@@ -29,11 +29,6 @@ disallow_untyped_defs = False
 disallow_incomplete_defs = False
 check_untyped_defs = False
 
-[mypy-qapi.types]
-disallow_untyped_defs = False
-disallow_incomplete_defs = False
-check_untyped_defs = False
-
 [mypy-qapi.visit]
 disallow_untyped_defs = False
 disallow_incomplete_defs = False
diff --git a/scripts/qapi/types.py b/scripts/qapi/types.py
index 53b47f9e58..766822feb3 100644
--- a/scripts/qapi/types.py
+++ b/scripts/qapi/types.py
@@ -13,6 +13,8 @@ This work is licensed under the terms of the GNU GPL, version 
2.
 # See the COPYING file in the top-level directory.
 """
 
+from typing import List, Optional
+
 from .common import (
 c_enum_const,
 c_name,
@@ -21,7 +23,16 @@ from .common import (
 mcgen,
 )
 from .gen import QAPISchemaModularCVisitor, ifcontext
-from .schema import QAPISchemaEnumMember, QAPISchemaObjectType
+from .schema import (
+QAPISchema,
+QAPISchemaEnumMember,
+QAPISchemaFeature,
+QAPISchemaObjectType,
+QAPISchemaObjectTypeMember,
+QAPISchemaType,
+QAPISchemaVariants,
+)
+from .source import QAPISourceInfo
 
 
 # variants must be emitted before their container; track what has already
@@ -29,7 +40,9 @@ from .schema import QAPISchemaEnumMember, QAPISchemaObjectType
 objects_seen = set()
 
 
-def gen_enum_lookup(name, members, prefix=None):
+def gen_enum_lookup(name: str,
+members: List[QAPISchemaEnumMember],
+prefix: Optional[str] = None) -> str:
 ret = mcgen('''
 
 const QEnumLookup %(c_name)s_lookup = {
@@ -54,7 +67,9 @@ const QEnumLookup %(c_name)s_lookup = {
 return ret
 
 
-def gen_enum(name, members, prefix=None):
+def gen_enum(name: str,
+ members: List[QAPISchemaEnumMember],
+ prefix: Optional[str] = None) -> str:
 # append automatically generated _MAX value
 enum_members = members + [QAPISchemaEnumMember('_MAX', None)]
 
@@ -88,7 +103,7 @@ extern const QEnumLookup %(c_name)s_lookup;
 return ret
 
 
-def gen_fwd_object_or_array(name):
+def gen_fwd_object_or_array(name: str) -> str:
 return mcgen('''
 
 typedef struct %(c_name)s %(c_name)s;
@@ -96,7 +111,7 @@ typedef struct %(c_name)s %(c_name)s;
  c_name=c_name(name))
 
 
-def gen_array(name, element_type):
+def gen_array(name: str, element_type: QAPISchemaType) -> str:
 return mcgen('''
 
 struct %(c_name)s {
@@ -107,7 +122,7 @@ struct %(c_name)s {
  c_name=c_name(name), c_type=element_type.c_type())
 
 
-def gen_struct_members(members):
+def gen_struct_members(members: List[QAPISchemaObjectTypeMember]) -> str:
 ret = ''
 for memb in members:
 ret += gen_if(memb.ifcond)
@@ -124,7 +139,10 @@ def gen_struct_members(members):
 return ret
 
 
-def gen_object(name, ifcond, base, members, variants):
+def gen_object(name: str, ifcond: List[str],
+   base: Optional[QAPISchemaObjectType],
+   members: List[QAPISchemaObjectTypeMember],
+   variants: Optional[QAPISchemaVariants]) -> str:
 if name in objects_seen:
 return ''
 objects_seen.add(name)
@@ -178,7 +196,7 @@ struct %(c_name)s {
 return ret
 
 
-def gen_upcast(name, base):
+def gen_upcast(name: str, base: QAPISchemaObjectType) -> str:
 # C makes const-correctness ugly.  We have to cast away const to let
 # this function work for both const and non-const obj.
 return mcgen('''
@@ -191,7 +209,7 @@ static inline %(base)s *qapi_%(c_name)s_base(const 
%(c_name)s *obj)
  c_name=c_name(name), base=base.c_name())
 
 
-def gen_variants(variants):
+def gen_variants(variants: QAPISchemaVariants) -> str:
 ret = mcgen('''
 union { /* union tag is @%(c_name)s */
 ''',
@@ -215,7 +233,7 @@ def gen_variants(variants):
 return ret
 
 
-def gen_type_cleanup_decl(name):
+def gen_type_cleanup_decl(name: str) -> str:
 ret = mcgen('''
 
 void qapi_free_%(c_name)s(%(c_name)s *obj);
@@ -225,7 +243,7 @@ G_DEFINE_AUTOPTR_CLEANUP_FUNC(%(c_name)s, 
qapi_free_%(c_name)s)
 return ret
 
 
-def gen_type_cleanup(name):
+def gen_type_cleanup(name: str) -> str:
 ret = mcgen('''
 
 void qapi_free_%(c_name)s(%(c_name)s *obj)
@@ -247,12 +265,12 @@ void qapi_free_%(c_name)s(%(c_name)s *obj)
 
 class QAPISchemaGenTypeVisitor(QAPISchemaModularCVisitor):
 
-def __init__(self, prefix):
+def __init__(se

[PULL 22/34] qapi/commands.py: add type hint annotations

[Markus Armbruster]

From: John Snow 

Annotations do not change runtime behavior.
This commit *only* adds annotations.

Signed-off-by: John Snow 
Reviewed-by: Eduardo Habkost 
Reviewed-by: Cleber Rosa 
Message-Id: <20201009161558.107041-23-js...@redhat.com>
Message-Id: <20201009161558.107041-24-js...@redhat.com>
[mypy.ini update squashed in]
Reviewed-by: Markus Armbruster 
Signed-off-by: Markus Armbruster 
---
 scripts/qapi/commands.py | 74 ++--
 scripts/qapi/mypy.ini|  5 ---
 2 files changed, 56 insertions(+), 23 deletions(-)

diff --git a/scripts/qapi/commands.py b/scripts/qapi/commands.py
index b0abb985a4..50978090b4 100644
--- a/scripts/qapi/commands.py
+++ b/scripts/qapi/commands.py
@@ -13,16 +13,34 @@ This work is licensed under the terms of the GNU GPL, 
version 2.
 See the COPYING file in the top-level directory.
 """
 
+from typing import (
+Dict,
+List,
+Optional,
+Set,
+)
+
 from .common import c_name, mcgen
 from .gen import (
+QAPIGenC,
 QAPIGenCCode,
 QAPISchemaModularCVisitor,
 build_params,
 ifcontext,
 )
+from .schema import (
+QAPISchema,
+QAPISchemaFeature,
+QAPISchemaObjectType,
+QAPISchemaType,
+)
+from .source import QAPISourceInfo
 
 
-def gen_command_decl(name, arg_type, boxed, ret_type):
+def gen_command_decl(name: str,
+ arg_type: Optional[QAPISchemaObjectType],
+ boxed: bool,
+ ret_type: Optional[QAPISchemaType]) -> str:
 return mcgen('''
 %(c_type)s qmp_%(c_name)s(%(params)s);
 ''',
@@ -31,7 +49,10 @@ def gen_command_decl(name, arg_type, boxed, ret_type):
  params=build_params(arg_type, boxed, 'Error **errp'))
 
 
-def gen_call(name, arg_type, boxed, ret_type):
+def gen_call(name: str,
+ arg_type: Optional[QAPISchemaObjectType],
+ boxed: bool,
+ ret_type: Optional[QAPISchemaType]) -> str:
 ret = ''
 
 argstr = ''
@@ -67,7 +88,7 @@ def gen_call(name, arg_type, boxed, ret_type):
 return ret
 
 
-def gen_marshal_output(ret_type):
+def gen_marshal_output(ret_type: QAPISchemaType) -> str:
 return mcgen('''
 
 static void qmp_marshal_output_%(c_name)s(%(c_type)s ret_in,
@@ -88,19 +109,22 @@ static void qmp_marshal_output_%(c_name)s(%(c_type)s 
ret_in,
  c_type=ret_type.c_type(), c_name=ret_type.c_name())
 
 
-def build_marshal_proto(name):
+def build_marshal_proto(name: str) -> str:
 return ('void qmp_marshal_%s(QDict *args, QObject **ret, Error **errp)'
 % c_name(name))
 
 
-def gen_marshal_decl(name):
+def gen_marshal_decl(name: str) -> str:
 return mcgen('''
 %(proto)s;
 ''',
  proto=build_marshal_proto(name))
 
 
-def gen_marshal(name, arg_type, boxed, ret_type):
+def gen_marshal(name: str,
+arg_type: Optional[QAPISchemaObjectType],
+boxed: bool,
+ret_type: Optional[QAPISchemaType]) -> str:
 have_args = boxed or (arg_type and not arg_type.is_empty())
 
 ret = mcgen('''
@@ -182,8 +206,11 @@ out:
 return ret
 
 
-def gen_register_command(name, success_response, allow_oob, allow_preconfig,
- coroutine):
+def gen_register_command(name: str,
+ success_response: bool,
+ allow_oob: bool,
+ allow_preconfig: bool,
+ coroutine: bool) -> str:
 options = []
 
 if not success_response:
@@ -207,7 +234,7 @@ def gen_register_command(name, success_response, allow_oob, 
allow_preconfig,
 return ret
 
 
-def gen_registry(registry, prefix):
+def gen_registry(registry: str, prefix: str) -> str:
 ret = mcgen('''
 
 void %(c_prefix)sqmp_init_marshal(QmpCommandList *cmds)
@@ -224,15 +251,14 @@ void %(c_prefix)sqmp_init_marshal(QmpCommandList *cmds)
 
 
 class QAPISchemaGenCommandVisitor(QAPISchemaModularCVisitor):
-
-def __init__(self, prefix):
+def __init__(self, prefix: str):
 super().__init__(
 prefix, 'qapi-commands',
 ' * Schema-defined QAPI/QMP commands', None, __doc__)
 self._regy = QAPIGenCCode(None)
-self._visited_ret_types = {}
+self._visited_ret_types: Dict[QAPIGenC, Set[QAPISchemaType]] = {}
 
-def _begin_user_module(self, name):
+def _begin_user_module(self, name: str) -> None:
 self._visited_ret_types[self._genc] = set()
 commands = self._module_basename('qapi-commands', name)
 types = self._module_basename('qapi-types', name)
@@ -256,7 +282,7 @@ class 
QAPISchemaGenCommandVisitor(QAPISchemaModularCVisitor):
 ''',
  types=types))
 
-def visit_end(self):
+def visit_end(self) -> None:
 self._add_system_module('init', ' * QAPI Commands initialization')
 self._genh.add(mcgen('''
 #include "qapi/qmp/dispatch.h"
@@ -272,9 +298,19 @@ void %(c_prefix)sqmp_init_marshal(QmpCommandList *cmds);
   

[PULL 33/34] qapi/visit.py: remove unused parameters from gen_visit_object

[Markus Armbruster]

From: John Snow 

And this fixes the pylint report for this file, so make sure we check
this in the future, too.

Signed-off-by: John Snow 
Reviewed-by: Eduardo Habkost 
Reviewed-by: Cleber Rosa 
Tested-by: Cleber Rosa 
Reviewed-by: Philippe Mathieu-Daudé 
Message-Id: <20201009161558.107041-36-js...@redhat.com>
Reviewed-by: Markus Armbruster 
Signed-off-by: Markus Armbruster 
---
 scripts/qapi/pylintrc | 1 -
 scripts/qapi/visit.py | 4 ++--
 2 files changed, 2 insertions(+), 3 deletions(-)

diff --git a/scripts/qapi/pylintrc b/scripts/qapi/pylintrc
index b3c4cf46db..b9e077a164 100644
--- a/scripts/qapi/pylintrc
+++ b/scripts/qapi/pylintrc
@@ -6,7 +6,6 @@ ignore-patterns=error.py,
 expr.py,
 parser.py,
 schema.py,
-visit.py,
 
 
 [MESSAGES CONTROL]
diff --git a/scripts/qapi/visit.py b/scripts/qapi/visit.py
index e00f2a09d7..8699e5c09c 100644
--- a/scripts/qapi/visit.py
+++ b/scripts/qapi/visit.py
@@ -250,7 +250,7 @@ out_obj:
 return ret
 
 
-def gen_visit_object(name, base, members, variants):
+def gen_visit_object(name):
 return mcgen('''
 
 bool visit_type_%(c_name)s(Visitor *v, const char *name,
@@ -343,7 +343,7 @@ class QAPISchemaGenVisitVisitor(QAPISchemaModularCVisitor):
 if not name.startswith('q_'):
 # only explicit types need an allocating visit
 self._genh.add(gen_visit_decl(name))
-self._genc.add(gen_visit_object(name, base, members, variants))
+self._genc.add(gen_visit_object(name))
 
 def visit_alternate_type(self, name, info, ifcond, features, variants):
 with ifcontext(ifcond, self._genh, self._genc):
-- 
2.26.2

[PULL 28/34] qapi/gen.py: update write() to be more idiomatic

[Markus Armbruster]

From: John Snow 

Make the file handling here just a tiny bit more idiomatic.
(I realize this is heavily subjective.)

Use exist_ok=True for os.makedirs and remove the exception,
use fdopen() to wrap the file descriptor in a File-like object,
and use a context manager for managing the file pointer.

Signed-off-by: John Snow 
Reviewed-by: Eduardo Habkost 
Reviewed-by: Cleber Rosa 
Reviewed-by: Markus Armbruster 
Message-Id: <20201009161558.107041-31-js...@redhat.com>
Signed-off-by: Markus Armbruster 
---
 scripts/qapi/gen.py | 25 +++--
 1 file changed, 11 insertions(+), 14 deletions(-)

diff --git a/scripts/qapi/gen.py b/scripts/qapi/gen.py
index 8b851c6262..670c21e210 100644
--- a/scripts/qapi/gen.py
+++ b/scripts/qapi/gen.py
@@ -12,7 +12,6 @@
 # See the COPYING file in the top-level directory.
 
 from contextlib import contextmanager
-import errno
 import os
 import re
 from typing import (
@@ -65,21 +64,19 @@ class QAPIGen:
 return
 pathname = os.path.join(output_dir, self.fname)
 odir = os.path.dirname(pathname)
+
 if odir:
-try:
-os.makedirs(odir)
-except os.error as e:
-if e.errno != errno.EEXIST:
-raise
+os.makedirs(odir, exist_ok=True)
+
+# use os.open for O_CREAT to create and read a non-existant file
 fd = os.open(pathname, os.O_RDWR | os.O_CREAT, 0o666)
-f = open(fd, 'r+', encoding='utf-8')
-text = self.get_content()
-oldtext = f.read(len(text) + 1)
-if text != oldtext:
-f.seek(0)
-f.truncate(0)
-f.write(text)
-f.close()
+with os.fdopen(fd, 'r+', encoding='utf-8') as fp:
+text = self.get_content()
+oldtext = fp.read(len(text) + 1)
+if text != oldtext:
+fp.seek(0)
+fp.truncate(0)
+fp.write(text)
 
 
 def _wrap_ifcond(ifcond: List[str], before: str, after: str) -> str:
-- 
2.26.2

[PULL 32/34] qapi/visit.py: assert tag_member contains a QAPISchemaEnumType

[Markus Armbruster]

From: John Snow 

This is true by design, but not presently able to be expressed in the
type system. An assertion helps mypy understand our constraints.

Signed-off-by: John Snow 
Reviewed-by: Eduardo Habkost 
Reviewed-by: Cleber Rosa 
Message-Id: <20201009161558.107041-35-js...@redhat.com>
Reviewed-by: Markus Armbruster 
Signed-off-by: Markus Armbruster 
---
 scripts/qapi/visit.py | 12 +++-
 1 file changed, 7 insertions(+), 5 deletions(-)

diff --git a/scripts/qapi/visit.py b/scripts/qapi/visit.py
index 708f72c4a1..e00f2a09d7 100644
--- a/scripts/qapi/visit.py
+++ b/scripts/qapi/visit.py
@@ -22,7 +22,7 @@ from .common import (
 mcgen,
 )
 from .gen import QAPISchemaModularCVisitor, ifcontext
-from .schema import QAPISchemaObjectType
+from .schema import QAPISchemaEnumType, QAPISchemaObjectType
 
 
 def gen_visit_decl(name, scalar=False):
@@ -84,15 +84,17 @@ bool visit_type_%(c_name)s_members(Visitor *v, %(c_name)s 
*obj, Error **errp)
 ret += gen_endif(memb.ifcond)
 
 if variants:
+tag_member = variants.tag_member
+assert isinstance(tag_member.type, QAPISchemaEnumType)
+
 ret += mcgen('''
 switch (obj->%(c_name)s) {
 ''',
- c_name=c_name(variants.tag_member.name))
+ c_name=c_name(tag_member.name))
 
 for var in variants.variants:
-case_str = c_enum_const(variants.tag_member.type.name,
-var.name,
-variants.tag_member.type.prefix)
+case_str = c_enum_const(tag_member.type.name, var.name,
+tag_member.type.prefix)
 ret += gen_if(var.ifcond)
 if var.type.name == 'q_empty':
 # valid variant and nothing to do
-- 
2.26.2

[PULL 19/34] qapi/events.py: add type hint annotations

[Markus Armbruster]

From: John Snow 

Annotations do not change runtime behavior.
This commit *only* adds annotations.

Note: __init__ does not need its return type annotated, as it is special.
https://mypy.readthedocs.io/en/stable/class_basics.html#annotating-init-methods

Signed-off-by: John Snow 
Reviewed-by: Eduardo Habkost 
Reviewed-by: Cleber Rosa 
Message-Id: <20201009161558.107041-20-js...@redhat.com>
Reviewed-by: Markus Armbruster 
Signed-off-by: Markus Armbruster 
---
 scripts/qapi/events.py | 46 --
 scripts/qapi/mypy.ini  |  5 -
 2 files changed, 35 insertions(+), 16 deletions(-)

diff --git a/scripts/qapi/events.py b/scripts/qapi/events.py
index f840a62ed9..57e0939e96 100644
--- a/scripts/qapi/events.py
+++ b/scripts/qapi/events.py
@@ -12,19 +12,31 @@ This work is licensed under the terms of the GNU GPL, 
version 2.
 See the COPYING file in the top-level directory.
 """
 
+from typing import List
+
 from .common import c_enum_const, c_name, mcgen
 from .gen import QAPISchemaModularCVisitor, build_params, ifcontext
-from .schema import QAPISchemaEnumMember
+from .schema import (
+QAPISchema,
+QAPISchemaEnumMember,
+QAPISchemaFeature,
+QAPISchemaObjectType,
+)
+from .source import QAPISourceInfo
 from .types import gen_enum, gen_enum_lookup
 
 
-def build_event_send_proto(name, arg_type, boxed):
+def build_event_send_proto(name: str,
+   arg_type: QAPISchemaObjectType,
+   boxed: bool) -> str:
 return 'void qapi_event_send_%(c_name)s(%(param)s)' % {
 'c_name': c_name(name.lower()),
 'param': build_params(arg_type, boxed)}
 
 
-def gen_event_send_decl(name, arg_type, boxed):
+def gen_event_send_decl(name: str,
+arg_type: QAPISchemaObjectType,
+boxed: bool) -> str:
 return mcgen('''
 
 %(proto)s;
@@ -33,7 +45,7 @@ def gen_event_send_decl(name, arg_type, boxed):
 
 
 # Declare and initialize an object 'qapi' using parameters from build_params()
-def gen_param_var(typ):
+def gen_param_var(typ: QAPISchemaObjectType) -> str:
 assert not typ.variants
 ret = mcgen('''
 %(c_name)s param = {
@@ -61,7 +73,11 @@ def gen_param_var(typ):
 return ret
 
 
-def gen_event_send(name, arg_type, boxed, event_enum_name, event_emit):
+def gen_event_send(name: str,
+   arg_type: QAPISchemaObjectType,
+   boxed: bool,
+   event_enum_name: str,
+   event_emit: str) -> str:
 # FIXME: Our declaration of local variables (and of 'errp' in the
 # parameter list) can collide with exploded members of the event's
 # data type passed in as parameters.  If this collision ever hits in
@@ -137,15 +153,15 @@ def gen_event_send(name, arg_type, boxed, 
event_enum_name, event_emit):
 
 class QAPISchemaGenEventVisitor(QAPISchemaModularCVisitor):
 
-def __init__(self, prefix):
+def __init__(self, prefix: str):
 super().__init__(
 prefix, 'qapi-events',
 ' * Schema-defined QAPI/QMP events', None, __doc__)
 self._event_enum_name = c_name(prefix + 'QAPIEvent', protect=False)
-self._event_enum_members = []
+self._event_enum_members: List[QAPISchemaEnumMember] = []
 self._event_emit_name = c_name(prefix + 'qapi_event_emit')
 
-def _begin_user_module(self, name):
+def _begin_user_module(self, name: str) -> None:
 events = self._module_basename('qapi-events', name)
 types = self._module_basename('qapi-types', name)
 visit = self._module_basename('qapi-visit', name)
@@ -168,7 +184,7 @@ class QAPISchemaGenEventVisitor(QAPISchemaModularCVisitor):
 ''',
  types=types))
 
-def visit_end(self):
+def visit_end(self) -> None:
 self._add_system_module('emit', ' * QAPI Events emission')
 self._genc.preamble_add(mcgen('''
 #include "qemu/osdep.h"
@@ -189,7 +205,13 @@ void %(event_emit)s(%(event_enum)s event, QDict *qdict);
  event_emit=self._event_emit_name,
  event_enum=self._event_enum_name))
 
-def visit_event(self, name, info, ifcond, features, arg_type, boxed):
+def visit_event(self,
+name: str,
+info: QAPISourceInfo,
+ifcond: List[str],
+features: List[QAPISchemaFeature],
+arg_type: QAPISchemaObjectType,
+boxed: bool) -> None:
 with ifcontext(ifcond, self._genh, self._genc):
 self._genh.add(gen_event_send_decl(name, arg_type, boxed))
 self._genc.add(gen_event_send(name, arg_type, boxed,
@@ -200,7 +222,9 @@ void %(event_emit)s(%(event_enum)s event, QDict *qdict);
 self._event_enum_members.append(QAPISchemaEnumMember(name, None))
 
 
-def gen_events(schema, output_dir, prefix):
+def gen_events(schema: QAPISchema,
+   outp

[PULL 31/34] qapi/types.py: remove one-letter variables

[Markus Armbruster]

From: John Snow 

"John, if pylint told you to jump off a bridge, would you?"
Hey, if it looked like fun, I might.

Now that this file is clean, enable pylint checks on this file.

Signed-off-by: John Snow 
Reviewed-by: Eduardo Habkost 
Reviewed-by: Cleber Rosa 
Message-Id: <20201009161558.107041-34-js...@redhat.com>
Reviewed-by: Markus Armbruster 
Signed-off-by: Markus Armbruster 
---
 scripts/qapi/pylintrc |  1 -
 scripts/qapi/types.py | 29 +++--
 2 files changed, 15 insertions(+), 15 deletions(-)

diff --git a/scripts/qapi/pylintrc b/scripts/qapi/pylintrc
index 8badcb11cd..b3c4cf46db 100644
--- a/scripts/qapi/pylintrc
+++ b/scripts/qapi/pylintrc
@@ -6,7 +6,6 @@ ignore-patterns=error.py,
 expr.py,
 parser.py,
 schema.py,
-types.py,
 visit.py,
 
 
diff --git a/scripts/qapi/types.py b/scripts/qapi/types.py
index 766822feb3..2b4916cdaa 100644
--- a/scripts/qapi/types.py
+++ b/scripts/qapi/types.py
@@ -49,14 +49,14 @@ const QEnumLookup %(c_name)s_lookup = {
 .array = (const char *const[]) {
 ''',
 c_name=c_name(name))
-for m in members:
-ret += gen_if(m.ifcond)
-index = c_enum_const(name, m.name, prefix)
+for memb in members:
+ret += gen_if(memb.ifcond)
+index = c_enum_const(name, memb.name, prefix)
 ret += mcgen('''
 [%(index)s] = "%(name)s",
 ''',
- index=index, name=m.name)
-ret += gen_endif(m.ifcond)
+ index=index, name=memb.name)
+ret += gen_endif(memb.ifcond)
 
 ret += mcgen('''
 },
@@ -79,13 +79,13 @@ typedef enum %(c_name)s {
 ''',
 c_name=c_name(name))
 
-for m in enum_members:
-ret += gen_if(m.ifcond)
+for memb in enum_members:
+ret += gen_if(memb.ifcond)
 ret += mcgen('''
 %(c_enum)s,
 ''',
- c_enum=c_enum_const(name, m.name, prefix))
-ret += gen_endif(m.ifcond)
+ c_enum=c_enum_const(name, memb.name, prefix))
+ret += gen_endif(memb.ifcond)
 
 ret += mcgen('''
 } %(c_name)s;
@@ -148,11 +148,12 @@ def gen_object(name: str, ifcond: List[str],
 objects_seen.add(name)
 
 ret = ''
-if variants:
-for v in variants.variants:
-if isinstance(v.type, QAPISchemaObjectType):
-ret += gen_object(v.type.name, v.type.ifcond, v.type.base,
-  v.type.local_members, v.type.variants)
+for var in variants.variants if variants else ():
+obj = var.type
+if not isinstance(obj, QAPISchemaObjectType):
+continue
+ret += gen_object(obj.name, obj.ifcond, obj.base,
+  obj.local_members, obj.variants)
 
 ret += mcgen('''
 
-- 
2.26.2

Re: [RFC PATCH 2/2] .mailmap: Fix more contributor entries

[Pan Nengyuan]

On 2020/10/5 1:26, Philippe Mathieu-Daudé wrote:
> These authors have some incorrect author email field.
> For each of them, there is one commit with the replaced
> entry.
> 
> Cc: Anup Patel 
> Cc: Andrew Melnychenko 
> Cc: David Carlier 
> Cc: Erik Smit 
> Cc: Marcel Apfelbaum 
> Cc: Pan Nengyuan 
> Cc: Stefan Berger 
> Signed-off-by: Philippe Mathieu-Daudé 
> ---

Acked-by: Pan Nengyuan 

Re: [RFC v1 3/4] qtest: do not build ide-test if TCG is not available

[Claudio Fontana]

On 10/9/20 6:01 PM, Paolo Bonzini wrote:
> On 09/10/20 17:21, Claudio Fontana wrote:
>> it seems that ide-test depends on TCG currently.
>>
>> Signed-off-by: Claudio Fontana 
>> ---
>>  tests/qtest/meson.build | 2 +-
>>  1 file changed, 1 insertion(+), 1 deletion(-)
>>
>> diff --git a/tests/qtest/meson.build b/tests/qtest/meson.build
>> index ad33ac311d..3418f65e2a 100644
>> --- a/tests/qtest/meson.build
>> +++ b/tests/qtest/meson.build
>> @@ -46,9 +46,9 @@ qtests_i386 = \
>>(config_all_devices.has_key('CONFIG_TPM_TIS_ISA') ? ['tpm-tis-test'] : 
>> []) +  \
>>(config_all_devices.has_key('CONFIG_TPM_TIS_ISA') ? 
>> ['tpm-tis-swtpm-test'] : []) +\
>>(config_all_devices.has_key('CONFIG_RTL8139_PCI') ? ['rtl8139-test'] : 
>> []) +  \
>> +  (config_all.has_key('CONFIG_TCG') ? ['ide-test'] : []) +  
>> \
>>qtests_pci +  
>> \
>>['fdc-test',
>> -   'ide-test',
>> 'hd-geo-test',
>> 'boot-order-test',
>> 'bios-tables-test',
>>
> 
> Interesting, why?...
> 
> Paolo
> 
> 

I am slowly trying to find out. I found out that the qos-test that buzzes is 
ide-test,
and I found out which specific ide test it was by manually bisecting functions 
inside the qtest_add_func in ide-test.c.

The issue seems limited to qtest_add_func("/ide/bmdma/trim", test_bmdma_trim);
No idea yet why that test buzzes forever.

Side note, maybe more verbose output on which specific test is attempted could 
be helpful? maybe only enabled on make V=2 ?

So the buzz.
top says:

22621 claudio   20   0   89700   3292   3004 R 53.82 0.010   1:22.43 ide-test   
   
22844 claudio   20   0 1026700  61168  38632 R 99.67 0.188   2:39.53 
qemu-system-i38   
25325 claudio   20   0   89700   3208   2940 R 52.16 0.010   0:56.05 ide-test   

25403 claudio   20   0 1026720  63028  38416 R 99.67 0.194   1:48.63 
qemu-system-x86   


i386 and x86_64 seem to show the exact same behaviour.


gdb says:

qemu-system-x86 (25403):

(gdb) info threads
  Id   Target Id   Frame 
* 1Thread 0x7fe35a406140 (LWP 25403) "qemu-system-x86" 0x7fe35157f7d6 
in ppoll () from /lib64/libc.so.6
  2Thread 0x7fe33946e700 (LWP 25415) "qemu-system-x86" 0x7fe351584839 
in syscall () from /lib64/libc.so.6
  3Thread 0x7fe338c6d700 (LWP 25439) "qemu-system-x86" 0x7fe35157f6db 
in poll () from /lib64/libc.so.6
  4Thread 0x7fe333fff700 (LWP 25440) "qemu-system-x86" 0x7fe35185bdcf 
in do_sigwait () from /lib64/libpthread.so.0

(gdb) thread 1
[Switching to thread 1 (Thread 0x7fe35a406140 (LWP 25403))]
#0  0x7fe35157f7d6 in ppoll () from /lib64/libc.so.6
(gdb) bt
#0  0x7fe35157f7d6 in ppoll () at /lib64/libc.so.6
#1  0x55a1f3138309 in ppoll (__ss=0x0, __timeout=0x7fff64d10b70, 
__nfds=, __fds=)
at /usr/include/bits/poll2.h:77
#2  0x55a1f3138309 in qemu_poll_ns (fds=, nfds=, timeout=timeout@entry=27462700)
at ../util/qemu-timer.c:349
#3  0x55a1f31512a5 in os_host_main_loop_wait (timeout=27462700) at 
../util/main-loop.c:239
#4  0x55a1f31512a5 in main_loop_wait (nonblocking=nonblocking@entry=0) at 
../util/main-loop.c:520
#5  0x55a1f2fc4bbd in qemu_main_loop () at ../softmmu/vl.c:1677
#6  0x55a1f2d001fe in main (argc=, argv=, 
envp=) at ../softmmu/main.c:50

(gdb) thread 2
[Switching to thread 2 (Thread 0x7fe33946e700 (LWP 25415))]
#0  0x7fe351584839 in syscall () from /lib64/libc.so.6
(gdb) bt
#0  0x7fe351584839 in syscall () at /lib64/libc.so.6
#1  0x55a1f312605b in qemu_futex_wait (val=, f=)
at /home/claudio/git/qemu-pristine/qemu/include/qemu/futex.h:29
#2  0x55a1f312605b in qemu_event_wait (ev=ev@entry=0x55a1f3a44208 
) at ../util/qemu-thread-posix.c:460
#3  0x55a1f314f868 in call_rcu_thread (opaque=opaque@entry=0x0) at 
../util/rcu.c:258
#4  0x55a1f3125276 in qemu_thread_start (args=) at 
../util/qemu-thread-posix.c:521
#5  0x7fe3518514f9 in start_thread () at /lib64/libpthread.so.0
#6  0x7fe351589fbf in clone () at /lib64/libc.so.6
(gdb) frame 3
#3  0x55a1f314f868 in call_rcu_thread (opaque=opaque@entry=0x0) at 
../util/rcu.c:258
258 qemu_event_wait(&rcu_call_ready_event);
(gdb) list 258
253 n = qatomic_read(&rcu_call_count);
254 if (n == 0) {
255 #if defined(CONFIG_MALLOC_TRIM)
256 malloc_trim(4 * 1024 * 1024);
257 #endif
258 qemu_event_wait(&rcu_call_ready_event);
259 }
260 }
261 n = qatomic_read(&rcu_call_count);
262 }


(gdb) thread 3
[Switching to thread 3 (Thread 0x7fe338c6d700 (LWP 25439))]
#0  0x7fe35157f6db in poll () from 

Re: [PATCH] net/filter-rewriter: destroy g_hash_table in colo_rewriter_cleanup

[Pan Nengyuan]

ping!

Maybe missed to queue?

On 2020/9/5 8:44, Li Qiang wrote:
> Pan Nengyuan  于2020年9月4日周五 下午3:23写道：
>>
>> s->connection_track_table forgot to destroy in colo_rewriter_cleanup. Fix it.
>>
>> Reported-by: Euler Robot 
>> Signed-off-by: Pan Nengyuan 
> 
> Reviewed-by: Li Qiang 
> 
>> ---
>>  net/filter-rewriter.c | 2 ++
>>  1 file changed, 2 insertions(+)
>>
>> diff --git a/net/filter-rewriter.c b/net/filter-rewriter.c
>> index 1aaad101b6..9ff366d44f 100644
>> --- a/net/filter-rewriter.c
>> +++ b/net/filter-rewriter.c
>> @@ -376,6 +376,8 @@ static void colo_rewriter_cleanup(NetFilterState *nf)
>>  filter_rewriter_flush(nf);
>>  g_free(s->incoming_queue);
>>  }
>> +
>> +g_hash_table_destroy(s->connection_track_table);
>>  }
>>
>>  static void colo_rewriter_setup(NetFilterState *nf, Error **errp)
>> --
>> 2.18.2
>>
>>

Re: [PATCH] target/i386/cpu: add return value verification and ignore Error objects

[Pan Nengyuan]

ping!

Maybe missed to queue?

On 2020/9/4 21:20, Philippe Mathieu-Daudé wrote:
> On 9/4/20 3:45 PM, Pan Nengyuan wrote:
>> 'err' is unnecessary in x86_cpu_class_check_missing_features(), we can 
>> change x86_cpu_expand_features()
>> to return true on success, false on failure, then pass NULL here to remove 
>> it.
>>
>> Signed-off-by: Pan Nengyuan 
>> Suggested-by: Markus Armbruster 
> 
> Reviewed-by: Philippe Mathieu-Daudé 
> 
>> ---
>>  target/i386/cpu.c | 15 +++
>>  1 file changed, 7 insertions(+), 8 deletions(-)
>>
>> diff --git a/target/i386/cpu.c b/target/i386/cpu.c
>> index 49d8958528..c3d3766133 100644
>> --- a/target/i386/cpu.c
>> +++ b/target/i386/cpu.c
>> @@ -4883,7 +4883,7 @@ static void x86_cpu_parse_featurestr(const char 
>> *typename, char *features,
>>  }
>>  }
>>  
>> -static void x86_cpu_expand_features(X86CPU *cpu, Error **errp);
>> +static bool x86_cpu_expand_features(X86CPU *cpu, Error **errp);
>>  static void x86_cpu_filter_features(X86CPU *cpu, bool verbose);
>>  
>>  /* Build a list with the name of all features on a feature word array */
>> @@ -4925,7 +4925,6 @@ static void 
>> x86_cpu_class_check_missing_features(X86CPUClass *xcc,
>>   strList **missing_feats)
>>  {
>>  X86CPU *xc;
>> -Error *err = NULL;
>>  strList **next = missing_feats;
>>  
>>  if (xcc->host_cpuid_required && !accel_uses_host_cpuid()) {
>> @@ -4937,8 +4936,7 @@ static void 
>> x86_cpu_class_check_missing_features(X86CPUClass *xcc,
>>  
>>  xc = X86_CPU(object_new_with_class(OBJECT_CLASS(xcc)));
>>  
>> -x86_cpu_expand_features(xc, &err);
>> -if (err) {
>> +if (!x86_cpu_expand_features(xc, NULL)) {
>>  /* Errors at x86_cpu_expand_features should never happen,
>>   * but in case it does, just report the model as not
>>   * runnable at all using the "type" property.
>> @@ -4947,7 +4945,6 @@ static void 
>> x86_cpu_class_check_missing_features(X86CPUClass *xcc,
>>  new->value = g_strdup("type");
>>  *next = new;
>>  next = &new->next;
>> -error_free(err);
>>  }
>>  
>>  x86_cpu_filter_features(xc, false);
>> @@ -6426,7 +6423,7 @@ static void x86_cpu_enable_xsave_components(X86CPU 
>> *cpu)
>>  /* Expand CPU configuration data, based on configured features
>>   * and host/accelerator capabilities when appropriate.
>>   */
>> -static void x86_cpu_expand_features(X86CPU *cpu, Error **errp)
>> +static bool x86_cpu_expand_features(X86CPU *cpu, Error **errp)
>>  {
>>  CPUX86State *env = &cpu->env;
>>  FeatureWord w;
>> @@ -6436,14 +6433,14 @@ static void x86_cpu_expand_features(X86CPU *cpu, 
>> Error **errp)
>>  for (l = plus_features; l; l = l->next) {
>>  const char *prop = l->data;
>>  if (!object_property_set_bool(OBJECT(cpu), prop, true, errp)) {
>> -return;
>> +return false;
>>  }
>>  }
>>  
>>  for (l = minus_features; l; l = l->next) {
>>  const char *prop = l->data;
>>  if (!object_property_set_bool(OBJECT(cpu), prop, false, errp)) {
>> -return;
>> +return false;
>>  }
>>  }
>>  
>> @@ -6540,6 +6537,8 @@ static void x86_cpu_expand_features(X86CPU *cpu, Error 
>> **errp)
>>  if (env->cpuid_xlevel2 == UINT32_MAX) {
>>  env->cpuid_xlevel2 = env->cpuid_min_xlevel2;
>>  }
>> +
>> +return true;
>>  }
>>  
>>  /*
>>
> 

Re: [PATCH v2 00/10] fix some error memleaks

[Pan Nengyuan]

ping!

Hello,

Some patches have been reviewed a few weeks ago but missed to queue.

The rest patches are the following:
https://patchwork.kernel.org/patch/11745621/
https://patchwork.kernel.org/patch/11745633/
https://patchwork.kernel.org/patch/11745629/
https://patchwork.kernel.org/patch/11745625/
https://patchwork.kernel.org/patch/11745627/
https://patchwork.kernel.org/patch/11745635/

On 2020/9/21 10:12, Pan Nengyuan wrote:
> ping!
> 
> and cc: qemu-triv...@nongnu.org
> 
> On 2020/9/17 20:49, Pan Nengyuan wrote:
>> ping!
>>
>> Anyone queued the rest(patch 01/02/03/07/08/09)?
>>
>> On 2020/8/31 21:43, Pan Nengyuan wrote:
>>> This series fix some Error/GError memleaks.
>>>
>>> V2: 
>>>   1. remove two patches.(One has aleardy applied. The other has fixed.)
>>>   2. change patch 5/10 and 7/10.
>>>
>>> Pan Nengyuan (10):
>>>   qga/channel-posix: Plug memory leak in ga_channel_write_all()
>>>   elf2dmp/qemu_elf: Plug memleak in QEMU_Elf_init
>>>   elf2dmp/pdb: Plug memleak in pdb_init_from_file
>>>   target/i386/sev: Plug memleak in sev_read_file_base64
>>>   ui/gtk-gl-area: Plug memleak in gd_gl_area_create_context()
>>>   target/i386/cpu: Fix memleak in x86_cpu_class_check_missing_features
>>>   migration/colo: Plug memleaks in colo_process_incoming_thread
>>>   blockdev: Fix a memleak in drive_backup_prepare()
>>>   block/file-posix: fix a possible undefined behavior
>>>   vnc-auth-sasl: Plug memleak in vnc_socket_ip_addr_string
>>>
>>>  block/file-posix.c |  2 +-
>>>  blockdev.c |  1 +
>>>  contrib/elf2dmp/pdb.c  |  1 +
>>>  contrib/elf2dmp/qemu_elf.c |  1 +
>>>  migration/colo.c   |  5 -
>>>  qga/channel-posix.c|  6 +-
>>>  target/i386/cpu.c  |  1 +
>>>  target/i386/sev.c  |  1 +
>>>  ui/gtk-gl-area.c   | 11 +++
>>>  ui/vnc-auth-sasl.c |  1 +
>>>  10 files changed, 27 insertions(+), 3 deletions(-)
>>>

[PATCH] migration/block-dirty-bitmap: fix uninitialized variable warning

[Chen Qun]

This if statement judgment is redundant and it will cause a warning:

migration/block-dirty-bitmap.c:1090:13: warning: ‘bitmap_name’ may be used
 uninitialized in this function [-Wmaybe-uninitialized]
 g_strlcpy(s->bitmap_name, bitmap_name, sizeof(s->bitmap_name));
 ^~

Reported-by: Euler Robot 
Signed-off-by: Chen Qun 
---
 migration/block-dirty-bitmap.c | 2 --
 1 file changed, 2 deletions(-)

diff --git a/migration/block-dirty-bitmap.c b/migration/block-dirty-bitmap.c
index 5bef793ac0..e09ea4f22b 100644
--- a/migration/block-dirty-bitmap.c
+++ b/migration/block-dirty-bitmap.c
@@ -1084,9 +1084,7 @@ static int dirty_bitmap_load_header(QEMUFile *f, 
DBMLoadState *s,
 } else {
 bitmap_name = s->bitmap_alias;
 }
-}
 
-if (!s->cancelled) {
 g_strlcpy(s->bitmap_name, bitmap_name, sizeof(s->bitmap_name));
 s->bitmap = bdrv_find_dirty_bitmap(s->bs, s->bitmap_name);
 
-- 
2.23.0

Re: [PATCH v2 10/15] hw/misc/bcm2835_cprman: add a clock mux skeleton implementation

[Luc Michel]

On 10:40 Tue 06 Oct , Philippe Mathieu-Daudé wrote:
> On 10/5/20 9:56 PM, Luc Michel wrote:
> > The clock multiplexers are the last clock stage in the CPRMAN. Each mux
> > outputs one clock signal that goes out of the CPRMAN to the SoC
> > peripherals.
> > 
> > Each mux has at most 10 sources. The sources 0 to 3 are common to all
> > muxes. They are:
> >0. ground (no clock signal)
> >1. the main oscillator (xosc)
> >2. "test debug 0" clock
> >3. "test debug 1" clock
> > 
> > Test debug 0 and 1 are actual clock muxes that can be used as sources to
> > other muxes (for debug purpose).
> > 
> > Sources 4 to 9 are mux specific and can be unpopulated (grounded). Those
> > sources are fed by the PLL channels outputs.
> > 
> > One corner case exists for DSI0E and DSI0P muxes. They have their source
> > number 4 connected to an intermediate multiplexer that can select
> > between PLLA-DSI0 and PLLD-DSI0 channel. This multiplexer is called
> > DSI0HSCK and is not a clock mux as such. It is really a simple mux from
> > the hardware point of view (see https://elinux.org/The_Undocumented_Pi).
> > This mux is not implemented in this commit.
> > 
> > Note that there is some muxes for which sources are unknown (because of
> > a lack of documentation). For those cases all the sources are connected
> > to ground in this implementation.
> > 
> > Each clock mux output is exported by the CPRMAN at the qdev level,
> > adding the suffix '-out' to the mux name to form the output clock name.
> > (E.g. the 'uart' mux sees its output exported as 'uart-out' at the
> > CPRMAN level.)
> > 
> > Tested-by: Philippe Mathieu-Daudé 
> > Signed-off-by: Luc Michel 
> > ---
> >  include/hw/misc/bcm2835_cprman.h   |  84 
> >  include/hw/misc/bcm2835_cprman_internals.h | 421 +
> >  hw/misc/bcm2835_cprman.c   | 150 
> >  3 files changed, 655 insertions(+)
> [...]
> 
> > +#define FILL_CLOCK_MUX_INIT_INFO(clock_, kind_) \
> > +.cm_offset = R_CM_ ## clock_ ## CTL,\
> > +FILL_CLOCK_MUX_SRC_MAPPING_INIT_INFO(kind_)
> > +
> > +static ClockMuxInitInfo CLOCK_MUX_INIT_INFO[] = {
> > +[CPRMAN_CLOCK_GNRIC] = {
> > +.name = "gnric",
> > +FILL_CLOCK_MUX_INIT_INFO(GNRIC, unknown),
> > +},
> [...]
> 
> > +static inline void update_mux_from_cm(BCM2835CprmanState *s, size_t idx)
> > +{
> > +size_t i;
> > +
> > +for (i = 0; i < CPRMAN_NUM_CLOCK_MUX; i++) {
> > +if ((CLOCK_MUX_INIT_INFO[i].cm_offset == idx) ||
> > +(CLOCK_MUX_INIT_INFO[i].cm_offset == idx + 4)) {
> 
> Maybe add a /* Matches DIV or CTL */ comment. Anyway
> FILL_CLOCK_MUX_INIT_INFO() only uses CTL, not DIV, so
> +4 check is not necessary.
I'll add a comment to clarify this. The +4 check is indeed necessary.
This comes from the fact that for all muxes, CM_CTL and CM_DIV registers are
contiguous. So I treat CprmanClockMuxState.reg_cm as a 2 cells array (as
I do for PLLs and the A2W_ANA regs). I'll clarify this point in the next
re-roll.

> 
> Otherwise:
> Reviewed-by: Philippe Mathieu-Daudé 
Thanks!
> 
> > +clock_mux_update(&s->clock_muxes[i]);
> > +return;
> > +}
> > +}
> > +}
> > +
> >  #define CASE_PLL_A2W_REGS(pll_) \
> >  case R_A2W_ ## pll_ ## _CTRL: \
> >  case R_A2W_ ## pll_ ## _ANA0: \
> >  case R_A2W_ ## pll_ ## _ANA1: \
> >  case R_A2W_ ## pll_ ## _ANA2: \
> > @@ -363,10 +438,19 @@ static void cprman_write(void *opaque, hwaddr offset,
> >  case R_A2W_PLLH_RCAL:
> >  case R_A2W_PLLH_PIX:
> >  case R_A2W_PLLB_ARM:
> >  update_channel_from_a2w(s, idx);
> >  break;
> > +
> > +case R_CM_GNRICCTL ... R_CM_SMIDIV:
> > +case R_CM_TCNTCNT ... R_CM_VECDIV:
> > +case R_CM_PULSECTL ... R_CM_PULSEDIV:
> > +case R_CM_SDCCTL ... R_CM_ARMCTL:
> > +case R_CM_AVEOCTL ... R_CM_EMMCDIV:
> > +case R_CM_EMMC2CTL ... R_CM_EMMC2DIV:
> > +update_mux_from_cm(s, idx);
> > +break;
> >  }
> >  }
> [...]

-- 

Re: [PATCH v2 10/15] hw/misc/bcm2835_cprman: add a clock mux skeleton implementation

[Luc Michel]

On 13:33 Sat 10 Oct , Luc Michel wrote:
> On 10:40 Tue 06 Oct , Philippe Mathieu-Daudé wrote:
> > On 10/5/20 9:56 PM, Luc Michel wrote:
> > > The clock multiplexers are the last clock stage in the CPRMAN. Each mux
> > > outputs one clock signal that goes out of the CPRMAN to the SoC
> > > peripherals.
> > > 
> > > Each mux has at most 10 sources. The sources 0 to 3 are common to all
> > > muxes. They are:
> > >0. ground (no clock signal)
> > >1. the main oscillator (xosc)
> > >2. "test debug 0" clock
> > >3. "test debug 1" clock
> > > 
> > > Test debug 0 and 1 are actual clock muxes that can be used as sources to
> > > other muxes (for debug purpose).
> > > 
> > > Sources 4 to 9 are mux specific and can be unpopulated (grounded). Those
> > > sources are fed by the PLL channels outputs.
> > > 
> > > One corner case exists for DSI0E and DSI0P muxes. They have their source
> > > number 4 connected to an intermediate multiplexer that can select
> > > between PLLA-DSI0 and PLLD-DSI0 channel. This multiplexer is called
> > > DSI0HSCK and is not a clock mux as such. It is really a simple mux from
> > > the hardware point of view (see https://elinux.org/The_Undocumented_Pi).
> > > This mux is not implemented in this commit.
> > > 
> > > Note that there is some muxes for which sources are unknown (because of
> > > a lack of documentation). For those cases all the sources are connected
> > > to ground in this implementation.
> > > 
> > > Each clock mux output is exported by the CPRMAN at the qdev level,
> > > adding the suffix '-out' to the mux name to form the output clock name.
> > > (E.g. the 'uart' mux sees its output exported as 'uart-out' at the
> > > CPRMAN level.)
> > > 
> > > Tested-by: Philippe Mathieu-Daudé 
> > > Signed-off-by: Luc Michel 
> > > ---
> > >  include/hw/misc/bcm2835_cprman.h   |  84 
> > >  include/hw/misc/bcm2835_cprman_internals.h | 421 +
> > >  hw/misc/bcm2835_cprman.c   | 150 
> > >  3 files changed, 655 insertions(+)
> > [...]
> > 
> > > +#define FILL_CLOCK_MUX_INIT_INFO(clock_, kind_) \
> > > +.cm_offset = R_CM_ ## clock_ ## CTL,\
> > > +FILL_CLOCK_MUX_SRC_MAPPING_INIT_INFO(kind_)
> > > +
> > > +static ClockMuxInitInfo CLOCK_MUX_INIT_INFO[] = {
> > > +[CPRMAN_CLOCK_GNRIC] = {
> > > +.name = "gnric",
> > > +FILL_CLOCK_MUX_INIT_INFO(GNRIC, unknown),
> > > +},
> > [...]
> > 
> > > +static inline void update_mux_from_cm(BCM2835CprmanState *s, size_t idx)
> > > +{
> > > +size_t i;
> > > +
> > > +for (i = 0; i < CPRMAN_NUM_CLOCK_MUX; i++) {
> > > +if ((CLOCK_MUX_INIT_INFO[i].cm_offset == idx) ||
> > > +(CLOCK_MUX_INIT_INFO[i].cm_offset == idx + 4)) {
> > 
> > Maybe add a /* Matches DIV or CTL */ comment. Anyway
> > FILL_CLOCK_MUX_INIT_INFO() only uses CTL, not DIV, so
> > +4 check is not necessary.
> I'll add a comment to clarify this. The +4 check is indeed necessary.
> This comes from the fact that for all muxes, CM_CTL and CM_DIV registers are
> contiguous. So I treat CprmanClockMuxState.reg_cm as a 2 cells array (as
> I do for PLLs and the A2W_ANA regs). I'll clarify this point in the next
> re-roll.
But the +4 is actually on the wrong side of the comparison :) I'll fix
that.
> 
> > 
> > Otherwise:
> > Reviewed-by: Philippe Mathieu-Daudé 
> Thanks!
> > 
> > > +clock_mux_update(&s->clock_muxes[i]);
> > > +return;
> > > +}
> > > +}
> > > +}
> > > +
> > >  #define CASE_PLL_A2W_REGS(pll_) \
> > >  case R_A2W_ ## pll_ ## _CTRL: \
> > >  case R_A2W_ ## pll_ ## _ANA0: \
> > >  case R_A2W_ ## pll_ ## _ANA1: \
> > >  case R_A2W_ ## pll_ ## _ANA2: \
> > > @@ -363,10 +438,19 @@ static void cprman_write(void *opaque, hwaddr 
> > > offset,
> > >  case R_A2W_PLLH_RCAL:
> > >  case R_A2W_PLLH_PIX:
> > >  case R_A2W_PLLB_ARM:
> > >  update_channel_from_a2w(s, idx);
> > >  break;
> > > +
> > > +case R_CM_GNRICCTL ... R_CM_SMIDIV:
> > > +case R_CM_TCNTCNT ... R_CM_VECDIV:
> > > +case R_CM_PULSECTL ... R_CM_PULSEDIV:
> > > +case R_CM_SDCCTL ... R_CM_ARMCTL:
> > > +case R_CM_AVEOCTL ... R_CM_EMMCDIV:
> > > +case R_CM_EMMC2CTL ... R_CM_EMMC2DIV:
> > > +update_mux_from_cm(s, idx);
> > > +break;
> > >  }
> > >  }
> > [...]
> 
> -- 

--