On 9/16/21 6:09 AM, John Snow wrote:
> Add a main() function to linters.py so that the Python CI infrastructure
> has something it can run.
>
> Now, linters.py represents an invocation of the linting scripts that
> more resembles a "normal" execution of pylint/mypy, like you'd expect to
> use if '
On 9/16/21 6:09 AM, John Snow wrote:
> Instead of running "run_linters" directly, create a main() function that
> will be responsible for environment setup, leaving run_linters()
> responsible only for execution of the linters.
>
> (That environment setup will be moved over in forthcoming commits.
On 9/16/21 6:09 AM, John Snow wrote:
> We need to import things from the qemu namespace; importing the
> namespace alone doesn't bring the submodules with it -- unless someone
> else (like iotests.py) imports them too.
>
> Adjust the imports.
>
> Signed-off-by: John Snow
> ---
> tests/qemu-iote
We can circumvent the '__main__' redefinition problem by passing
--scripts-are-modules. Take mypy out of the loop per-filename and check
everything in one go: it's quite a bit faster.
Signed-off-by: John Snow
---
tests/qemu-iotests/linters.py | 62 ---
1 file chan
This one is insidious: if you use the invocation
"from {namespace} import {subpackage}" as mirror-top-perms does,
mypy will fail on every-other invocation *if* the package being
imported is a package.
Now, I could just edit mirror-top-perms to avoid this invocation, but
since I tripped on a landmi
Split the linter execution itself out from 297, leaving just environment
setup in 297. This is done so that non-iotest code can invoke the
linters without needing to worry about imports of unpackaged iotest
code.
Eventually, it should be possible to replace linters.py with mypy.ini
and pylintrc fi
As a convenience, since iotests is an extremely prominent user of the
qemu.qmp and qemu.machine packages and already implements a linting
regime, run those tests as well so that it's very hard to miss
regressions caused by changes to the python library.
Signed-off-by: John Snow
---
python/tests/
On 9/16/21 6:09 AM, John Snow wrote:
> Mostly uninteresting stuff. Move the test injections under a function
> named main() so that the variables used during that process aren't in
> the global scope.
>
> Signed-off-by: John Snow
> ---
> tests/qemu-iotests/tests/migrate-bitmaps-test | 50 +++
'pylint-3' is another Fedora-ism. Use "python3 -m pylint" or "python3 -m
mypy" to access these scripts instead. This style of invocation will
prefer the "correct" tool when run in a virtual environment.
Note that we still check for "pylint-3" before the test begins -- this
check is now "overly str
Allow run_linters to work well if it's executed from a different
directory.
Signed-off-by: John Snow
Reviewed-by: Vladimir Sementsov-Ogievskiy
---
tests/qemu-iotests/297 | 3 +++
1 file changed, 3 insertions(+)
diff --git a/tests/qemu-iotests/297 b/tests/qemu-iotests/297
index 08d2b87108..e05c
Add a main() function to linters.py so that the Python CI infrastructure
has something it can run.
Now, linters.py represents an invocation of the linting scripts that
more resembles a "normal" execution of pylint/mypy, like you'd expect to
use if 'qemu' was a bona-fide package you obtained from P
Add a directory argument to is_python_file to allow it to work correctly
no matter what CWD we happen to run it from. This is done in
anticipation of running the iotests from another directory (./python/).
Signed-off-by: John Snow
---
tests/qemu-iotests/297 | 8 +---
1 file changed, 5 insert
Instead of running "run_linters" directly, create a main() function that
will be responsible for environment setup, leaving run_linters()
responsible only for execution of the linters.
(That environment setup will be moved over in forthcoming commits.)
Signed-off-by: John Snow
Reviewed-by: Vladi
This turns run_linters() into a bit of a hybrid test; returning non-zero
on failed execution while also printing diffable information. This is
done for the benefit of the avocado simple test runner, which will soon
be attempting to execute this test from a different environment.
(Note: universal_n
Mostly uninteresting stuff. Move the test injections under a function
named main() so that the variables used during that process aren't in
the global scope.
Signed-off-by: John Snow
---
tests/qemu-iotests/tests/migrate-bitmaps-test | 50 +++
1 file changed, 28 insertions(+), 22
Signed-off-by: John Snow
---
tests/qemu-iotests/tests/migrate-bitmaps-postcopy-test | 3 +++
1 file changed, 3 insertions(+)
diff --git a/tests/qemu-iotests/tests/migrate-bitmaps-postcopy-test
b/tests/qemu-iotests/tests/migrate-bitmaps-postcopy-test
index 00ebb5c251..9c00ae61c8 100755
--- a/tes
GitLab: https://gitlab.com/jsnow/qemu/-/commits/python-package-iotest
CI: https://gitlab.com/jsnow/qemu/-/pipelines/371611883
Based-On: <20210915175318.853225-1-hre...@redhat.com>
"[PULL 00/32] Block patches"
Since iotests are such a heavy and prominent user of the Python qemu.qmp
and qe
Split out file discovery into its own method to begin separating out the
"environment setup" and "test execution" phases.
Signed-off-by: John Snow
---
tests/qemu-iotests/297 | 12 +---
1 file changed, 9 insertions(+), 3 deletions(-)
diff --git a/tests/qemu-iotests/297 b/tests/qemu-iotes
A few new annoyances. Of note is the new warning for an unspecified
encoding when opening a text file, which actually does indicate a
potentially real problem; see
https://www.python.org/dev/peps/pep-0597/#motivation
Use LC_CTYPE to determine an encoding to use for interpreting QEMU's
terminal out
Move environment setup into main(), leaving pure test execution behind
in run_linters().
Signed-off-by: John Snow
Reviewed-by: Vladimir Sementsov-Ogievskiy
---
tests/qemu-iotests/297 | 36 +---
1 file changed, 21 insertions(+), 15 deletions(-)
diff --git a/tests
We need to import things from the qemu namespace; importing the
namespace alone doesn't bring the submodules with it -- unless someone
else (like iotests.py) imports them too.
Adjust the imports.
Signed-off-by: John Snow
---
tests/qemu-iotests/tests/mirror-top-perms | 7 ---
1 file changed,
ping & Cc'ing qemu-trivial@ (reviewed twice) ...
On 8/16/21 8:04 PM, Philippe Mathieu-Daudé wrote:
> When configuring QEMU with --disable-fuse, the qemu-storage-daemon
> still reports FUSE command line options in its help:
>
> $ qemu-storage-daemon -h
> Usage: qemu-storage-daemon [options]
>
Markus Armbruster writes:
> Vladimir Sementsov-Ogievskiy writes:
>
>> 08.06.2021 14:12, Markus Armbruster wrote:
>>> Vladimir Sementsov-Ogievskiy writes:
>>>
>>> [...]
>>>
TODO: We also need to deprecate drive-backup transaction action..
But union members in QAPI doesn't support 'de
From: Vladimir Sementsov-Ogievskiy
Split checking for reserved bits out of aligned offset check.
Signed-off-by: Vladimir Sementsov-Ogievskiy
Reviewed-by: Eric Blake
Tested-by: Kirill Tkhai
Reviewed-by: Hanna Reitz
Message-Id: <20210914122454.141075-11-vsement...@virtuozzo.com>
Signed-off-by:
From: Vladimir Sementsov-Ogievskiy
- use g_autofree for l1_table
- better name for size in bytes variable
- reduce code blocks nesting
- whitespaces, braces, newlines
Signed-off-by: Vladimir Sementsov-Ogievskiy
Reviewed-by: Hanna Reitz
Message-Id: <20210914122454.141075-9-vsement...@virtuo
From: Vladimir Sementsov-Ogievskiy
Signed-off-by: Vladimir Sementsov-Ogievskiy
Reviewed-by: Eric Blake
Tested-by: Kirill Tkhai
Reviewed-by: Hanna Reitz
Message-Id: <20210914122454.141075-10-vsement...@virtuozzo.com>
Signed-off-by: Hanna Reitz
---
block/qcow2.h | 1 +
block/qcow2-re
From: Vladimir Sementsov-Ogievskiy
We'll reuse the function to fix wrong L2 entry bitmap. Support it now.
Signed-off-by: Vladimir Sementsov-Ogievskiy
Reviewed-by: Eric Blake
Reviewed-by: Hanna Reitz
Message-Id: <20210914122454.141075-6-vsement...@virtuozzo.com>
Signed-off-by: Hanna Reitz
---
From: Vladimir Sementsov-Ogievskiy
Check subcluster bitmap of the l2 entry for different types of
clusters:
- for compressed it must be zero
- for allocated check consistency of two parts of the bitmap
- for unallocated all subclusters should be unallocated
(or zero-plain)
Signed-off-by:
We cannot write to images opened with O_DIRECT unless we allow them to
be resized so they are aligned to the sector size: Since 9c60a5d1978,
bdrv_node_refresh_perm() ensures that for nodes whose length is not
aligned to the request alignment and where someone has taken a WRITE
permission, the RESIZ
From: Vladimir Sementsov-Ogievskiy
Split fix_l2_entry_by_zero() out of check_refcounts_l2() to be
reused in further patch.
Signed-off-by: Vladimir Sementsov-Ogievskiy
Reviewed-by: Eric Blake
Reviewed-by: Hanna Reitz
Message-Id: <20210914122454.141075-5-vsement...@virtuozzo.com>
Signed-off-by:
From: Vladimir Sementsov-Ogievskiy
Add helper to parse compressed l2_entry and use it everywhere instead
of open-coding.
Note, that in most places we move to precise coffset/csize instead of
sector-aligned. Still it should work good enough for updating
refcounts.
Signed-off-by: Vladimir Sements
From: Vladimir Sementsov-Ogievskiy
- don't use same name for size in bytes and in entries
- use g_autofree for l2_table
- add whitespace
- fix block comment style
Signed-off-by: Vladimir Sementsov-Ogievskiy
Reviewed-by: Eric Blake
Reviewed-by: Hanna Reitz
Message-Id: <20210914122454.14107
From: Vladimir Sementsov-Ogievskiy
There is no conflict and no dependency if we have parallel writes to
different subclusters of one cluster when the cluster itself is already
allocated. So, relax extra dependency.
Measure performance:
First, prepare build/qemu-img-old and build/qemu-img-new ima
From: Vladimir Sementsov-Ogievskiy
Add simple grammar-parsing template benchmark. New tool consume test
template written in bash with some special grammar injections and
produces multiple tests, run them and finally print a performance
comparison table of different tests produced from one templat
From: Vladimir Sementsov-Ogievskiy
No logic change, just prepare for the following commit. While being
here do also small grammar fix in a comment.
Signed-off-by: Vladimir Sementsov-Ogievskiy
Reviewed-by: Eric Blake
Reviewed-by: Hanna Reitz
Message-Id: <20210824101517.59802-3-vsement...@virtu
297 so far does not check the named tests, which reside in the tests/
directory (i.e. full path tests/qemu-iotests/tests). Fix it.
Thanks to the previous two commits, all named tests pass its scrutiny,
so we do not have to add anything to SKIP_FILES.
Signed-off-by: Hanna Reitz
Reviewed-by: Will
From: Eric Blake
Although we have long supported 'qemu-img convert -o
backing_file=foo,backing_fmt=bar', the fact that we have a shortcut -B
for backing_file but none for backing_fmt has made it more likely that
users accidentally run into:
qemu-img: warning: Deprecated use of backing file witho
From: Vladimir Sementsov-Ogievskiy
Signed-off-by: Vladimir Sementsov-Ogievskiy
Reviewed-by: Eric Blake
Tested-by: Kirill Tkhai
Reviewed-by: Hanna Reitz
Message-Id: <20210914122454.141075-8-vsement...@virtuozzo.com>
[hreitz: Separated `type` declaration from statements]
Signed-off-by: Hanna Re
From: Vladimir Sementsov-Ogievskiy
Let's pass the whole L2 entry and not bother with
L2E_COMPRESSED_OFFSET_SIZE_MASK.
It also helps further refactoring that adds generic
qcow2_parse_compressed_l2_entry() helper.
Signed-off-by: Vladimir Sementsov-Ogievskiy
Reviewed-by: Eric Blake
Reviewed-by:
From: Vladimir Sementsov-Ogievskiy
We must not inactivate child when parent has write permissions on
it.
Calling .bdrv_inactivate() doesn't help: actually only qcow2 has this
handler and it is used to flush caches, not for permission
manipulations.
So, let's simply check cumulative parent permi
From: Vladimir Sementsov-Ogievskiy
Add a simple test which tries to run migration during backup.
bdrv_inactivate_all() should fail. But due to bug (see next commit with
fix) it doesn't, nodes are inactivated and continued backup crashes
on assertion "assert(!(bs->open_flags & BDRV_O_INACTIVE));"
From: Stefano Garzarella
In mirror_iteration() we call mirror_wait_on_conflicts() with
`self` parameter set to NULL.
Starting from commit d44dae1a7c we dereference `self` pointer in
mirror_wait_on_conflicts() without checks if it is not NULL.
Backtrace:
Program terminated with signal SIGSEGV,
As of recently, pylint complains when `open()` calls are missing an
`encoding=` specified. Everything we have should be UTF-8 (and in fact,
everything should be UTF-8, period (exceptions apply)), so use that.
Signed-off-by: Hanna Reitz
Message-Id: <20210824153540.177128-2-hre...@redhat.com>
Revi
The AbnormalShutdown exception class is not in qemu.machine, but in
qemu.machine.machine. (qemu.machine.AbnormalShutdown was enough for
Python to find it in order to run this test, but pylint complains about
it.)
Signed-off-by: Hanna Reitz
Message-Id: <20210902094017.32902-5-hre...@redhat.com>
R
There are a couple of things pylint takes issue with:
- The "time" import is unused
- The import order (iotests should come last)
- get_bitmap_hash() doesn't use @self and so should be a function
- Semicolons at the end of some lines
- Parentheses after "if"
- Some lines are too long (80 characters
169 and 199 have been renamed and moved to tests/ (commit a44be0334be:
"iotests: rename and move 169 and 199 tests"), so we can drop them from
the skip list.
Signed-off-by: Hanna Reitz
Reviewed-by: Willian Rampazzo
Reviewed-by: Vladimir Sementsov-Ogievskiy
Reviewed-by: Kevin Wolf
Message-Id: <
bdrv_co_block_status() does it for us, we do not need to do it here.
The advantage of not capping *pnum is that bdrv_co_block_status() can
cache larger data regions than requested by its caller.
Signed-off-by: Hanna Reitz
Reviewed-by: Eric Blake
Reviewed-by: Vladimir Sementsov-Ogievskiy
Review
.bdrv_co_block_status() implementations are free to return a *pnum that
exceeds @bytes, because bdrv_co_block_status() in block/io.c will clamp
*pnum as necessary.
On the other hand, if drivers' implementations return values for *pnum
that are as large as possible, our recently introduced block-st
bdrv_co_block_status() does it for us, we do not need to do it here.
The advantage of not capping *pnum is that bdrv_co_block_status() can
cache larger data regions than requested by its caller.
Signed-off-by: Hanna Reitz
Reviewed-by: Eric Blake
Reviewed-by: Vladimir Sementsov-Ogievskiy
Review
There is a comment above the BDS definition stating care must be taken
to consider handling newly added fields in bdrv_append().
Actually, this comment should have said "bdrv_swap()" as of 4ddc07cac
(nine years ago), and in any case, bdrv_swap() was dropped in
8e419aefa (six years ago). So no suc
The following changes since commit 0b6206b9c6825619cd721085fe082d7a0abc9af4:
Merge remote-tracking branch 'remotes/rth-gitlab/tags/pull-tcg-20210914-4'
into staging (2021-09-15 13:27:49 +0100)
are available in the Git repository at:
https://github.com/XanClic/qemu.git tags/pull-block-2021-0
pylint complains that discards1_sha256 and all_discards_sha256 are first
set in non-__init__ methods.
These variables are not really class-variables anyway, so let them
instead be returned by start_postcopy(), thus silencing pylint.
Suggested-by: Vladimir Sementsov-Ogievskiy
Signed-off-by: Hanna
pylint proposes using `[]` instead of `list()` and `{}` instead of
`dict()`, because it is faster. That seems simple enough, so heed its
advice.
Signed-off-by: Hanna Reitz
Message-Id: <20210824153540.177128-3-hre...@redhat.com>
Reviewed-by: Vladimir Sementsov-Ogievskiy
---
tests/qemu-iotests/i
bdrv_co_block_status() does it for us, we do not need to do it here.
The advantage of not capping *pnum is that bdrv_co_block_status() can
cache larger data regions than requested by its caller.
Signed-off-by: Hanna Reitz
Reviewed-by: Eric Blake
Reviewed-by: Vladimir Sementsov-Ogievskiy
Review
As we have attempted before
(https://lists.gnu.org/archive/html/qemu-devel/2019-01/msg06451.html,
"file-posix: Cache lseek result for data regions";
https://lists.nongnu.org/archive/html/qemu-block/2021-02/msg00934.html,
"file-posix: Cache next hole"), this patch seeks to reduce the number of
SEEK_
From: Max Reitz
gluster's block-status implementation is basically a copy of that in
block/file-posix.c, there is only one thing missing, and that is
aligning trailing data extents to the request alignment (as added by
commit 9c3db310ff0).
Note that 9c3db310ff0 mentions that "there seems to be n
On Wed, Sep 15, 2021 at 05:43:30PM +0200, Pankaj Raghav wrote:
> Currently, FUSED operations are not supported by QEMU. As per the 1.4 SPEC,
> controller should abort the command that requested a fused operation with
> an INVALID FIELD error code if they are not supported.
>
> Changes from v1:
>
Currently, FUSED operations are not supported by QEMU. As per the 1.4 SPEC,
controller should abort the command that requested a fused operation with
an INVALID FIELD error code if they are not supported.
Changes from v1:
Added FUSE flag check also to the admin cmd processing as the FUSED
operat
https://bugzilla.redhat.com/show_bug.cgi?id=2004120
qemu-6.1.0-7.fc36.aarch64
Host kernel: 5.14.0-60.fc35.aarch64
Guest kernel: 5.14.1-300.fc35.armv7hl+lpae
Under I/O load, I get intermittent write errors inside the Fedora
armv7 guest running on aarch64 host.
[ 2063.691215] print_req_error: 6 ca
[Copying an email previously sent here:
https://lists.fedoraproject.org/archives/list/de...@lists.fedoraproject.org/thread/U6C6TTITVJVQFXUMUCPYB5PKXT2NMENI/#ZRVNBUMQ26GF3WSGDEVU2YAFIJFJS3QS]
It seems like the qemu test suite now hangs intermittently after
running iotests. Here are some examples:
On 14.09.21 12:25, Vladimir Sementsov-Ogievskiy wrote:
These series makes tests pass with
IMGOPTS='compression_type=zstd'
Also, python iotests start to support IMGOPTS (they didn't before).
Unfortunately, the problem I have now is that it makes the tests fail
with other IMGOPTS.
My reg
On Wed, Sep 15, 2021 at 02:11:41PM +0200, Paolo Bonzini wrote:
> On 13/09/21 15:10, Stefan Hajnoczi wrote:
> > On Wed, Sep 08, 2021 at 09:10:17AM -0400, Emanuele Giuseppe Esposito wrote:
> > > Currently, block layer APIs like block-backend.h contain a mix of
> > > functions that are either running
On Wed, Sep 15, 2021 at 02:14:08PM +0200, Paolo Bonzini wrote:
> On 13/09/21 15:27, Stefan Hajnoczi wrote:
> > > +/*
> > > + * I/O API functions. These functions are thread-safe, and therefore
> > > + * can run in any AioContext.
> > "can run in any AioContext" makes me wonder what the exact requir
On 13/09/21 15:27, Stefan Hajnoczi wrote:
+/*
+ * I/O API functions. These functions are thread-safe, and therefore
+ * can run in any AioContext.
"can run in any AioContext" makes me wonder what the exact requirements
are. Can they run in any*thread* (regardless of whether an AioContext
even e
On 13/09/21 15:10, Stefan Hajnoczi wrote:
On Wed, Sep 08, 2021 at 09:10:17AM -0400, Emanuele Giuseppe Esposito wrote:
Currently, block layer APIs like block-backend.h contain a mix of
functions that are either running in the main loop and under the
BQL, or are thread-safe functions and run in io
15.09.2021 13:26, Hanna Reitz wrote:
On 14.09.21 14:24, Vladimir Sementsov-Ogievskiy wrote:
Hi all!
Here are some good refactorings and new (qemu-img check) checks for
qcow2.
Thanks, applied to my block branch:
https://gitlab.com/hreitz/qemu/-/commits/block/
(Patch 6 is here:
https://gitla
On 13.09.21 15:17, Eric Blake wrote:
Although we have long supported 'qemu-img convert -o
backing_file=foo,backing_fmt=bar', the fact that we have a shortcut -B
for backing_file but none for backing_fmt has made it more likely that
users accidentally run into:
qemu-img: warning: Deprecated use o
On 14.09.21 14:24, Vladimir Sementsov-Ogievskiy wrote:
Hi all!
Here are some good refactorings and new (qemu-img check) checks for
qcow2.
Thanks, applied to my block branch:
https://gitlab.com/hreitz/qemu/-/commits/block/
(Patch 6 is here:
https://gitlab.com/hreitz/qemu/-/commit/93c40e7dab2
15.09.2021 12:00, Richard W.M. Jones wrote:
On Wed, Sep 15, 2021 at 10:15:20AM +0300, Vladimir Sementsov-Ogievskiy wrote:
14.09.2021 19:32, Richard W.M. Jones wrote:
On Tue, Sep 14, 2021 at 06:21:58PM +0300, Vladimir Sementsov-Ogievskiy wrote:
14.09.2021 17:52, Richard W.M. Jones wrote:
On th
On Wed, Sep 15, 2021 at 10:15:20AM +0300, Vladimir Sementsov-Ogievskiy wrote:
> 14.09.2021 19:32, Richard W.M. Jones wrote:
> >On Tue, Sep 14, 2021 at 06:21:58PM +0300, Vladimir Sementsov-Ogievskiy wrote:
> >>14.09.2021 17:52, Richard W.M. Jones wrote:
> >>>On the
> >>>server side when the server r
On Mon, 2021-09-13 at 08:17 -0500, Eric Blake wrote:
> Although we have long supported 'qemu-img convert -o
> backing_file=foo,backing_fmt=bar', the fact that we have a shortcut -B
> for backing_file but none for backing_fmt has made it more likely that
> users accidentally run into:
>
> qemu-img:
13.09.2021 16:17, Eric Blake wrote:
Although we have long supported 'qemu-img convert -o
backing_file=foo,backing_fmt=bar', the fact that we have a shortcut -B
for backing_file but none for backing_fmt has made it more likely that
users accidentally run into:
qemu-img: warning: Deprecated use of
07.09.2021 15:42, Hanna Reitz wrote:
Hi,
v1 cover letter:
https://lists.nongnu.org/archive/html/qemu-block/2021-07/msg00705.html
v2 cover letter:
https://lists.nongnu.org/archive/html/qemu-block/2021-07/msg00747.html
v3 cover letter:
https://lists.nongnu.org/archive/html/qemu-block/2021-08/msg
14.09.2021 19:32, Richard W.M. Jones wrote:
On Tue, Sep 14, 2021 at 06:21:58PM +0300, Vladimir Sementsov-Ogievskiy wrote:
14.09.2021 17:52, Richard W.M. Jones wrote:
On the
server side when the server receives NBD_CMD_DISC it must complete any
in-flight requests, but there's no requirement for
14.09.2021 20:20, Hanna Reitz wrote:
On 08.09.21 18:33, Vladimir Sementsov-Ogievskiy wrote:
07.09.2021 15:42, Hanna Reitz wrote:
Callers should be able to specify whether they want job_cancel_sync() to
force-cancel the job or not.
In fact, almost all invocations do not care about consistency o
14.09.2021 20:15, Hanna Reitz wrote:
On 14.09.21 14:24, Vladimir Sementsov-Ogievskiy wrote:
Signed-off-by: Vladimir Sementsov-Ogievskiy
Reviewed-by: Eric Blake
Tested-by: Kirill Tkhai
Reviewed-by: Hanna Reitz
---
block/qcow2.h | 1 +
block/qcow2-refcount.c | 12 +++-
2
76 matches
Mail list logo
