Re: [PATCH RFC 00/13] hw/nvme: experimental user-creatable objects

2021-09-17 Thread Kevin Wolf
Am 17.09.2021 um 08:21 hat Klaus Jensen geschrieben: > On Sep 16 18:30, Klaus Jensen wrote: > > On Sep 16 14:41, Kevin Wolf wrote: > > > Am 14.09.2021 um 22:37 hat Klaus Jensen geschrieben: > > > > From: Klaus Jensen > > > > > > > > Hi, > > > > > > > > This is an attempt at adressing a bunch of

Re: [PATCH RFC 00/13] hw/nvme: experimental user-creatable objects

2021-09-17 Thread Klaus Jensen
On Sep 17 09:38, Kevin Wolf wrote: > Am 17.09.2021 um 08:21 hat Klaus Jensen geschrieben: > > On Sep 16 18:30, Klaus Jensen wrote: > > > On Sep 16 14:41, Kevin Wolf wrote: > > > > Am 14.09.2021 um 22:37 hat Klaus Jensen geschrieben: > > > > > From: Klaus Jensen > > > > > > > > > > Hi, > > > > >

Re: [PATCH v3 02/16] iotests/mirror-top-perms: Adjust imports

2021-09-17 Thread Hanna Reitz
On 16.09.21 06:09, John Snow wrote: We need to import things from the qemu namespace; importing the namespace alone doesn't bring the submodules with it -- unless someone else (like iotests.py) imports them too. Adjust the imports. Signed-off-by: John Snow --- tests/qemu-iotests/tests/mirror

Re: [PATCH v3 03/16] iotests/migrate-bitmaps-postcopy-test: declare instance variables

2021-09-17 Thread Hanna Reitz
On 16.09.21 06:09, John Snow wrote: Signed-off-by: John Snow --- tests/qemu-iotests/tests/migrate-bitmaps-postcopy-test | 3 +++ 1 file changed, 3 insertions(+) diff --git a/tests/qemu-iotests/tests/migrate-bitmaps-postcopy-test b/tests/qemu-iotests/tests/migrate-bitmaps-postcopy-test index

Re: [PATCH v3 04/16] iotests/migrate-bitmaps-test: delint

2021-09-17 Thread Hanna Reitz
On 16.09.21 06:09, John Snow wrote: Mostly uninteresting stuff. Move the test injections under a function named main() so that the variables used during that process aren't in the global scope. Signed-off-by: John Snow --- tests/qemu-iotests/tests/migrate-bitmaps-test | 50 +++

Re: [PATCH v3 05/16] iotests/297: modify is_python_file to work from any CWD

2021-09-17 Thread Hanna Reitz
On 16.09.21 06:09, John Snow wrote: Add a directory argument to is_python_file to allow it to work correctly no matter what CWD we happen to run it from. This is done in anticipation of running the iotests from another directory (./python/). “the iotests” or just 297?  All of the iotests would

Re: [PATCH v3 06/16] iotests/297: Add get_files() function

2021-09-17 Thread Hanna Reitz
On 16.09.21 06:09, John Snow wrote: Split out file discovery into its own method to begin separating out the "environment setup" and "test execution" phases. Signed-off-by: John Snow --- tests/qemu-iotests/297 | 12 +--- 1 file changed, 9 insertions(+), 3 deletions(-) diff --git a/t

Re: [PATCH v3 07/16] iotests/297: Don't rely on distro-specific linter binaries

2021-09-17 Thread Hanna Reitz
On 16.09.21 06:09, John Snow wrote: 'pylint-3' is another Fedora-ism. Use "python3 -m pylint" or "python3 -m mypy" to access these scripts instead. This style of invocation will prefer the "correct" tool when run in a virtual environment. Note that we still check for "pylint-3" before the test b

Re: [PATCH v3 08/16] iotests/297: Create main() function

2021-09-17 Thread Hanna Reitz
On 16.09.21 06:09, John Snow wrote: Instead of running "run_linters" directly, create a main() function that will be responsible for environment setup, leaving run_linters() responsible only for execution of the linters. (That environment setup will be moved over in forthcoming commits.) Signed

Re: [PATCH v3 09/16] iotests/297: Separate environment setup from test execution

2021-09-17 Thread Hanna Reitz
On 16.09.21 06:09, John Snow wrote: Move environment setup into main(), leaving pure test execution behind in run_linters(). Signed-off-by: John Snow Reviewed-by: Vladimir Sementsov-Ogievskiy --- tests/qemu-iotests/297 | 36 +--- 1 file changed, 21 insertions

Re: [PATCH v3 10/16] iotests/297: Add 'directory' argument to run_linters

2021-09-17 Thread Hanna Reitz
On 16.09.21 06:09, John Snow wrote: Allow run_linters to work well if it's executed from a different directory. Signed-off-by: John Snow Reviewed-by: Vladimir Sementsov-Ogievskiy --- tests/qemu-iotests/297 | 3 +++ 1 file changed, 3 insertions(+) Reviewed-by: Hanna Reitz

Re: [PATCH v3 11/16] iotests/297: return error code from run_linters()

2021-09-17 Thread Hanna Reitz
On 16.09.21 06:09, John Snow wrote: This turns run_linters() into a bit of a hybrid test; returning non-zero on failed execution while also printing diffable information. This is done for the benefit of the avocado simple test runner, which will soon be attempting to execute this test from a diff

Re: [PATCH v3 14/16] iotests/linters: Add workaround for mypy bug #9852

2021-09-17 Thread Hanna Reitz
On 16.09.21 06:09, John Snow wrote: This one is insidious: if you use the invocation "from {namespace} import {subpackage}" as mirror-top-perms does, mypy will fail on every-other invocation *if* the package being imported is a package. Now, I could just edit mirror-top-perms to avoid this invoc

Re: [PATCH v3 16/16] iotests/linters: check mypy files all at once

2021-09-17 Thread Hanna Reitz
On 16.09.21 06:09, John Snow wrote: We can circumvent the '__main__' redefinition problem by passing --scripts-are-modules. Take mypy out of the loop per-filename and check everything in one go: it's quite a bit faster. Is it possible to pull this to the beginning of the series?  Just because

Re: [PATCH 01/15] python/aqmp: add greeting property to QMPClient

2021-09-17 Thread Hanna Reitz
On 17.09.21 07:40, John Snow wrote: Expose the greeting as a read-only property of QMPClient so it can be retrieved at-will. Signed-off-by: John Snow --- python/qemu/aqmp/qmp_client.py | 5 + 1 file changed, 5 insertions(+) Reviewed-by: Hanna Reitz

Re: [PATCH 02/15] python/aqmp: add .empty() method to EventListener

2021-09-17 Thread Hanna Reitz
On 17.09.21 07:40, John Snow wrote: Synchronous clients may want to know if they're about to block waiting for an event or not. A method such as this is necessary to implement a compatible interface for the old QEMUMonitorProtocol using the new async internals. Signed-off-by: John Snow --- py

Re: [PATCH 03/15] python/aqmp: Return cleared events from EventListener.clear()

2021-09-17 Thread Hanna Reitz
On 17.09.21 07:40, John Snow wrote: This serves two purposes: (1) It is now possible to discern whether or not clear() removed any event(s) from the queue with absolute certainty, and (2) It is now very easy to get a List of all pending events in one chunk, which is useful for the sync bridge.

Re: [PATCH 04/15] python/qmp: clear events on get_events() call

2021-09-17 Thread Hanna Reitz
On 17.09.21 07:40, John Snow wrote: All callers in the tree *already* clear the events after a call to get_events(). Do it automatically instead and update callsites to remove the manual clear call. These semantics are quite a bit easier to emulate with async QMP, and nobody appears to be abusin

Re: [PATCH 05/15] python/qmp: add send_fd_scm directly to QEMUMonitorProtocol

2021-09-17 Thread Hanna Reitz
On 17.09.21 07:40, John Snow wrote: It turns out you can do this directly from Python ... and because of this, you don't need to worry about setting the inheritability of the fds or spawning another process. Doing this is helpful because it allows QEMUMonitorProtocol to keep its file descriptor

Re: [PATCH 06/15] python, iotests: remove socket_scm_helper

2021-09-17 Thread Hanna Reitz
On 17.09.21 07:40, John Snow wrote: It's not used anymore, now. Signed-off-by: John Snow --- tests/qemu-iotests/socket_scm_helper.c | 136 - python/qemu/machine/machine.py | 3 - python/qemu/machine/qtest.py | 2 - tests/Makefile.include

Re: [PATCH 07/15] python/aqmp: add send_fd_scm

2021-09-17 Thread Hanna Reitz
On 17.09.21 07:40, John Snow wrote: The single space is indeed required to successfully transmit the file descriptor to QEMU. Yeah, socket_scm_helper.c said “Send a blank to notify qemu”. Signed-off-by: John Snow --- python/qemu/aqmp/qmp_client.py | 17 + 1 file changed, 1

Re: [PATCH 08/15] python/aqmp: Create MessageModel and StandaloneModel classes

2021-09-17 Thread Hanna Reitz
On 17.09.21 07:40, John Snow wrote: This allows 'Greeting' to be subclass of 'Message'. We need the adapter classes to avoid some typing problems that occur if we try to put too much into the 'Model' class itself; the exact details of why are left as an exercise to the reader. Why bother? This m

Re: [PATCH 09/15] python/machine: remove has_quit argument

2021-09-17 Thread Hanna Reitz
On 17.09.21 07:40, John Snow wrote: If we spy on the QMP commands instead, we don't need callers to remember to pass it. Seems like a fair trade-off. The one slightly weird bit is overloading this instance variable for wait(), where we use it to mean "don't issue the qmp 'quit' command". This me

Re: [PATCH 10/15] python/machine: Add support for AQMP backend

2021-09-17 Thread Hanna Reitz
On 17.09.21 07:40, John Snow wrote: To use the AQMP backend, Machine just needs to be a little more diligent about what happens when closing a QMP connection. The operation is no longer a freebie in the async world. Because async QMP continues to check for messages asynchronously, it's almost ce

Re: [PATCH 11/15] python/aqmp: Create sync QMP wrapper for iotests

2021-09-17 Thread Hanna Reitz
On 17.09.21 07:40, John Snow wrote: This is a wrapper around the async QMPClient that mimics the old, synchronous QEMUMonitorProtocol class. It is designed to be interchangeable with the old implementation. It does not, however, attempt to mimic Exception compatibility. Signed-off-by: John Snow

Re: [PATCH 12/15] iotests: Disable AQMP logging under non-debug modes

2021-09-17 Thread Hanna Reitz
On 17.09.21 07:40, John Snow wrote: Disable the aqmp logger, which likes to (at the moment) print out intermediate warnings and errors that cause session termination; disable them so they don't interfere with the job output. Leave any "CRITICAL" warnings enabled though, those are ones that we sh

Re: [PATCH 13/15] iotests: Accommodate async QMP Exception classes

2021-09-17 Thread Hanna Reitz
On 17.09.21 07:40, John Snow wrote: (But continue to support the old ones for now, too.) There are very few cases of any user of QEMUMachine or a subclass thereof relying on a QMP Exception type. If you'd like to check for yourself, you want to grep for all of the derivatives of QMPError, exclud

Re: [PATCH 14/15] python/aqmp: Remove scary message

2021-09-17 Thread Hanna Reitz
On 17.09.21 07:40, John Snow wrote: The scary message interferes with the iotests output. Coincidentally, if iotests works by removing this, then it's good evidence that we don't really need to scare people away from using it. Signed-off-by: John Snow --- python/qemu/aqmp/__init__.py | 14 ---

Re: [PATCH 15/15] python, iotests: replace qmp with aqmp

2021-09-17 Thread Hanna Reitz
On 17.09.21 07:40, John Snow wrote: Swap out the synchronous QEMUMonitorProtocol from qemu.qmp with the sync wrapper from qemu.aqmp instead. Add an escape hatch in the form of the environment variable QEMU_PYTHON_LEGACY_QMP which allows you to cajole QEMUMachine into using the old interface, pro

Re: [PATCH v3 00/17] iotests: support zstd

2021-09-17 Thread Hanna Reitz
On 15.09.21 16:45, Hanna Reitz wrote: On 14.09.21 12:25, Vladimir Sementsov-Ogievskiy wrote: These series makes tests pass with     IMGOPTS='compression_type=zstd' Also, python iotests start to support IMGOPTS (they didn't before). Unfortunately, the problem I have now is that it makes the t

Re: [PATCH 15/15] python, iotests: replace qmp with aqmp

2021-09-17 Thread John Snow
On Fri, Sep 17, 2021 at 10:40 AM Hanna Reitz wrote: > On 17.09.21 07:40, John Snow wrote: > > Swap out the synchronous QEMUMonitorProtocol from qemu.qmp with the sync > > wrapper from qemu.aqmp instead. > > > > Add an escape hatch in the form of the environment variable > > QEMU_PYTHON_LEGACY_QMP

Re: [PATCH 14/15] python/aqmp: Remove scary message

2021-09-17 Thread John Snow
On Fri, Sep 17, 2021 at 10:39 AM Hanna Reitz wrote: > On 17.09.21 07:40, John Snow wrote: > > The scary message interferes with the iotests output. Coincidentally, if > > iotests works by removing this, then it's good evidence that we don't > > really need to scare people away from using it. > >

Re: [PATCH v2] nbd/server: Suppress Broken pipe errors on abrupt disconnection

2021-09-17 Thread Eric Blake
On Wed, Sep 15, 2021 at 10:00:25AM +0100, Richard W.M. Jones wrote: > > >I would say the spec is at best contradictory, but if you read other > > >parts of the spec, then it's pretty clear that we're allowed to drop > > >the connection whenever we like. This section says as much: > > > > > >https:

Re: [PATCH v2] nbd/server: Suppress Broken pipe errors on abrupt disconnection

2021-09-17 Thread Eric Blake
On Wed, Sep 15, 2021 at 12:11:35PM +0300, Vladimir Sementsov-Ogievskiy wrote: > > > >There are two methods of terminating the transmission phase: > > > >... > > > >"The client or the server drops the TCP session (in which case it > > > >SHOULD shut down the TLS session first). This

Re: [PATCH 03/15] python/aqmp: Return cleared events from EventListener.clear()

2021-09-17 Thread John Snow
On Fri, Sep 17, 2021 at 8:36 AM Hanna Reitz wrote: > On 17.09.21 07:40, John Snow wrote: > > This serves two purposes: > > > > (1) It is now possible to discern whether or not clear() removed any > > event(s) from the queue with absolute certainty, and > > > > (2) It is now very easy to get a Lis

Re: [PATCH 04/15] python/qmp: clear events on get_events() call

2021-09-17 Thread John Snow
On Fri, Sep 17, 2021 at 8:51 AM Hanna Reitz wrote: > On 17.09.21 07:40, John Snow wrote: > > All callers in the tree *already* clear the events after a call to > > get_events(). Do it automatically instead and update callsites to remove > > the manual clear call. > > > > These semantics are quite

Re: [PATCH 05/15] python/qmp: add send_fd_scm directly to QEMUMonitorProtocol

2021-09-17 Thread John Snow
On Fri, Sep 17, 2021 at 9:21 AM Hanna Reitz wrote: > On 17.09.21 07:40, John Snow wrote: > > It turns out you can do this directly from Python ... and because of > > this, you don't need to worry about setting the inheritability of the > > fds or spawning another process. > > > > Doing this is he

Re: [PATCH 07/15] python/aqmp: add send_fd_scm

2021-09-17 Thread John Snow
On Fri, Sep 17, 2021 at 9:34 AM Hanna Reitz wrote: > On 17.09.21 07:40, John Snow wrote: > > The single space is indeed required to successfully transmit the file > > descriptor to QEMU. > > Yeah, socket_scm_helper.c said “Send a blank to notify qemu”. > > > Signed-off-by: John Snow > > --- > >

Re: [PATCH v3 00/17] iotests: support zstd

2021-09-17 Thread Vladimir Sementsov-Ogievskiy
17.09.2021 17:49, Hanna Reitz wrote: On 15.09.21 16:45, Hanna Reitz wrote: On 14.09.21 12:25, Vladimir Sementsov-Ogievskiy wrote: These series makes tests pass with     IMGOPTS='compression_type=zstd' Also, python iotests start to support IMGOPTS (they didn't before). Unfortunately, the pro

Re: [PATCH 08/15] python/aqmp: Create MessageModel and StandaloneModel classes

2021-09-17 Thread John Snow
On Fri, Sep 17, 2021, 9:39 AM Hanna Reitz wrote: > On 17.09.21 07:40, John Snow wrote: > > This allows 'Greeting' to be subclass of 'Message'. We need the adapter > > classes to avoid some typing problems that occur if we try to put too > > much into the 'Model' class itself; the exact details of

Re: [PATCH] qemu-img: Add -F shorthand to convert

2021-09-17 Thread Eric Blake
On Mon, Sep 13, 2021 at 08:17:35AM -0500, Eric Blake wrote: > Although we have long supported 'qemu-img convert -o > backing_file=foo,backing_fmt=bar', the fact that we have a shortcut -B > for backing_file but none for backing_fmt has made it more likely that > users accidentally run into: > > qe

Re: [PATCH 09/15] python/machine: remove has_quit argument

2021-09-17 Thread John Snow
On Fri, Sep 17, 2021 at 9:59 AM Hanna Reitz wrote: > On 17.09.21 07:40, John Snow wrote: > > If we spy on the QMP commands instead, we don't need callers to remember > > to pass it. Seems like a fair trade-off. > > > > The one slightly weird bit is overloading this instance variable for > > wait(

Re: [PATCH] qemu-img: Add -F shorthand to convert

2021-09-17 Thread Vladimir Sementsov-Ogievskiy
17.09.2021 23:13, Eric Blake wrote: On Mon, Sep 13, 2021 at 08:17:35AM -0500, Eric Blake wrote: Although we have long supported 'qemu-img convert -o backing_file=foo,backing_fmt=bar', the fact that we have a shortcut -B for backing_file but none for backing_fmt has made it more likely that users

Re: [PATCH 10/15] python/machine: Add support for AQMP backend

2021-09-17 Thread John Snow
On Fri, Sep 17, 2021 at 10:16 AM Hanna Reitz wrote: > On 17.09.21 07:40, John Snow wrote: > > To use the AQMP backend, Machine just needs to be a little more diligent > > about what happens when closing a QMP connection. The operation is no > > longer a freebie in the async world. > > > > Because

Re: [PATCH 11/15] python/aqmp: Create sync QMP wrapper for iotests

2021-09-17 Thread John Snow
On Fri, Sep 17, 2021 at 10:23 AM Hanna Reitz wrote: > On 17.09.21 07:40, John Snow wrote: > > This is a wrapper around the async QMPClient that mimics the old, > > synchronous QEMUMonitorProtocol class. It is designed to be > > interchangeable with the old implementation. > > > > It does not, how

Re: [PATCH 12/15] iotests: Disable AQMP logging under non-debug modes

2021-09-17 Thread John Snow
On Fri, Sep 17, 2021 at 10:30 AM Hanna Reitz wrote: > On 17.09.21 07:40, John Snow wrote: > > Disable the aqmp logger, which likes to (at the moment) print out > > intermediate warnings and errors that cause session termination; disable > > them so they don't interfere with the job output. > > >

Re: [PATCH 13/15] iotests: Accommodate async QMP Exception classes

2021-09-17 Thread John Snow
On Fri, Sep 17, 2021 at 10:35 AM Hanna Reitz wrote: > On 17.09.21 07:40, John Snow wrote: > > (But continue to support the old ones for now, too.) > > > > There are very few cases of any user of QEMUMachine or a subclass > > thereof relying on a QMP Exception type. If you'd like to check for > >

Re: [PATCH 12/15] iotests: Disable AQMP logging under non-debug modes

2021-09-17 Thread John Snow
On Fri, Sep 17, 2021 at 8:58 PM John Snow wrote: > > > On Fri, Sep 17, 2021 at 10:30 AM Hanna Reitz wrote: > >> On 17.09.21 07:40, John Snow wrote: >> > Disable the aqmp logger, which likes to (at the moment) print out >> > intermediate warnings and errors that cause session termination; disable