Re: [PATCH v5 04/20] job.c: move inner aiocontext lock in callbacks

2022-02-24 Thread Emanuele Giuseppe Esposito
On 17/02/2022 14:45, Stefan Hajnoczi wrote: > On Tue, Feb 08, 2022 at 09:34:57AM -0500, Emanuele Giuseppe Esposito wrote: >> Instead of having the lock in job_tnx_apply, move it inside > > s/tnx/txn/ > >> in the callback. This will be helpful for next commits, when >> we introduce job_lock/unl

Re: [PATCH v5 06/20] jobs: remove aiocontext locks since the functions are under BQL

2022-02-24 Thread Emanuele Giuseppe Esposito
On 17/02/2022 15:12, Stefan Hajnoczi wrote: > On Tue, Feb 08, 2022 at 09:34:59AM -0500, Emanuele Giuseppe Esposito wrote: >> In preparation to the job_lock/unlock patch, remove these >> aiocontext locks. >> The main reason these two locks are removed here is because >> they are inside a loop ite

Re: [PATCH v5 04/20] job.c: move inner aiocontext lock in callbacks

2022-02-24 Thread Emanuele Giuseppe Esposito
On 17/02/2022 14:45, Stefan Hajnoczi wrote: > On Tue, Feb 08, 2022 at 09:34:57AM -0500, Emanuele Giuseppe Esposito wrote: >> Instead of having the lock in job_tnx_apply, move it inside > > s/tnx/txn/ > >> in the callback. This will be helpful for next commits, when >> we introduce job_lock/unl

Re: [PATCH 1/3] util & iothread: Introduce event-loop abstract class

2022-02-24 Thread Stefan Hajnoczi
On Mon, Feb 21, 2022 at 06:08:43PM +0100, Nicolas Saenz Julienne wrote: > diff --git a/qom/meson.build b/qom/meson.build > index 062a3789d8..c20e5dd1cb 100644 > --- a/qom/meson.build > +++ b/qom/meson.build > @@ -4,6 +4,7 @@ qom_ss.add(files( >'object.c', >'object_interfaces.c', >'qom-q

Re: [PATCH v5 07/20] job.h: add _locked duplicates for job API functions called with and without job_mutex

2022-02-24 Thread Emanuele Giuseppe Esposito
On 17/02/2022 15:20, Stefan Hajnoczi wrote: > On Tue, Feb 08, 2022 at 09:35:00AM -0500, Emanuele Giuseppe Esposito wrote: >> static void job_exit(void *opaque) >> { >> Job *job = (Job *)opaque; >> AioContext *ctx; >> +JOB_LOCK_GUARD(); >> >> job_ref(job); >> aio_conte

Re: [PATCH 2/3] util/main-loop: Introduce the main loop into QOM

2022-02-24 Thread Stefan Hajnoczi
On Mon, Feb 21, 2022 at 06:08:44PM +0100, Nicolas Saenz Julienne wrote: > diff --git a/include/qemu/main-loop.h b/include/qemu/main-loop.h > index 8dbc6fcb89..fea5a3e9d4 100644 > --- a/include/qemu/main-loop.h > +++ b/include/qemu/main-loop.h > @@ -26,9 +26,20 @@ > #define QEMU_MAIN_LOOP_H > >

Re: [PATCH 3/3] util/event-loop: Introduce options to set the thread pool size

2022-02-24 Thread Stefan Hajnoczi
On Mon, Feb 21, 2022 at 06:08:45PM +0100, Nicolas Saenz Julienne wrote: > The thread pool regulates itself: when idle, it kills threads until > empty, when in demand, it creates new threads until full. This behaviour > doesn't play well with latency sensitive workloads where the price of > creating

Re: [PATCH v4 03/18] block/block-copy: block_copy_state_new(): add bitmap parameter

2022-02-24 Thread Hanna Reitz
On 16.02.22 20:46, Vladimir Sementsov-Ogievskiy wrote: This will be used in the following commit to bring "incremental" mode to copy-before-write filter. Signed-off-by: Vladimir Sementsov-Ogievskiy --- include/block/block-copy.h | 1 + block/block-copy.c | 14 +- block/

Re: [PATCH v4 04/18] block/copy-before-write: add bitmap open parameter

2022-02-24 Thread Hanna Reitz
On 16.02.22 20:46, Vladimir Sementsov-Ogievskiy wrote: This brings "incremental" mode to copy-before-write filter: user can specify bitmap so that filter will copy only "dirty" areas. Signed-off-by: Vladimir Sementsov-Ogievskiy --- qapi/block-core.json | 10 +++- block/copy-before-w

Re: [PATCH v4 06/18] block: intoduce reqlist

2022-02-24 Thread Hanna Reitz
On 16.02.22 20:46, Vladimir Sementsov-Ogievskiy wrote: Split intersecting-requests functionality out of block-copy to be reused in copy-before-write filter. Note: while being here, fix tiny typo in MAINTAINERS. Signed-off-by: Vladimir Sementsov-Ogievskiy --- include/block/reqlist.h | 67 +++

Re: [PATCH v4 08/18] block/dirty-bitmap: introduce bdrv_dirty_bitmap_status()

2022-02-24 Thread Hanna Reitz
On 16.02.22 20:46, Vladimir Sementsov-Ogievskiy wrote: Add a convenient function similar with bdrv_block_status() to get status of dirty bitmap. Signed-off-by: Vladimir Sementsov-Ogievskiy --- include/block/dirty-bitmap.h | 2 ++ include/qemu/hbitmap.h | 12 block/dirty-

Re: [PATCH v4 07/18] block/reqlist: reqlist_find_conflict(): use ranges_overlap()

2022-02-24 Thread Hanna Reitz
On 16.02.22 20:46, Vladimir Sementsov-Ogievskiy wrote: Let's reuse convenient helper. Signed-off-by: Vladimir Sementsov-Ogievskiy --- block/reqlist.c | 3 ++- 1 file changed, 2 insertions(+), 1 deletion(-) Reviewed-by: Hanna Reitz

Re: [PATCH v4 10/18] block/io: introduce block driver snapshot-access API

2022-02-24 Thread Hanna Reitz
On 16.02.22 20:46, Vladimir Sementsov-Ogievskiy wrote: Add new block driver handlers and corresponding generic wrappers. It will be used to allow copy-before-write filter to provide reach fleecing interface in further commit. In future this approach may be used to allow reading qcow2 interanal

Re: [PATCH v4 11/18] block: introduce snapshot-access filter

2022-02-24 Thread Hanna Reitz
On 16.02.22 20:46, Vladimir Sementsov-Ogievskiy wrote: The filter simply utilizes snapshot-access API of underlying block Nit picking: Well, it isn’t really a filter.  I understand where you’re coming from, but by definition it isn’t a filter driver. node. In further patches we want to use

Re: [PATCH v5 09/20] jobs: add job lock in find_* functions

2022-02-24 Thread Emanuele Giuseppe Esposito
On 17/02/2022 16:00, Stefan Hajnoczi wrote: > On Tue, Feb 08, 2022 at 09:35:02AM -0500, Emanuele Giuseppe Esposito wrote: >> diff --git a/blockdev.c b/blockdev.c >> index c5fba4d157..08408cd44b 100644 >> --- a/blockdev.c >> +++ b/blockdev.c >> @@ -3311,7 +3311,10 @@ out: >> aio_context_rele

Re: [PATCH v4 12/18] block: copy-before-write: realize snapshot-access API

2022-02-24 Thread Hanna Reitz
On 16.02.22 20:46, Vladimir Sementsov-Ogievskiy wrote: Current scheme of image fleecing looks like this: [guest][NBD export] | | |root | root v v [copy-before-write] -> [temp.qcow2

Re: [PULL 00/10] Misc next patches

2022-02-24 Thread Peter Maydell
On Mon, 21 Feb 2022 at 19:18, Daniel P. Berrangé wrote: > AFAIK, the block jobs run in CI don't cover the SSH driver at all. > > I had a CI pipeline before submitting, which covered Free BSD 13 > which passed. To be sure I just rebased to git master and tried > another pipeline which passed too: >

Re: [PATCH v5 08/20] jobs: protect jobs with job_lock/unlock

2022-02-24 Thread Emanuele Giuseppe Esposito
On 17/02/2022 15:48, Stefan Hajnoczi wrote: > On Tue, Feb 08, 2022 at 09:35:01AM -0500, Emanuele Giuseppe Esposito wrote: >> diff --git a/block/replication.c b/block/replication.c >> index 55c8f894aa..a03b28726e 100644 >> --- a/block/replication.c >> +++ b/block/replication.c >> @@ -149,7 +149,9

Re: [PATCH v4 15/18] iotests/image-fleecing: add test case with bitmap

2022-02-24 Thread Hanna Reitz
On 16.02.22 20:46, Vladimir Sementsov-Ogievskiy wrote: Note that reads zero areas (not dirty in the bitmap) fails, that's correct. Signed-off-by: Vladimir Sementsov-Ogievskiy --- tests/qemu-iotests/tests/image-fleecing | 32 ++-- tests/qemu-iotests/tests/image-fleecing.out | 84 +

Re: [PATCH v4 13/18] iotests/image-fleecing: add test-case for fleecing format node

2022-02-24 Thread Hanna Reitz
On 16.02.22 20:46, Vladimir Sementsov-Ogievskiy wrote: Signed-off-by: Vladimir Sementsov-Ogievskiy --- tests/qemu-iotests/tests/image-fleecing | 64 - tests/qemu-iotests/tests/image-fleecing.out | 76 - 2 files changed, 120 insertions(+), 20 deletions(

Re: [PATCH v4 14/18] iotests.py: add qemu_io_pipe_and_status()

2022-02-24 Thread Hanna Reitz
On 16.02.22 20:46, Vladimir Sementsov-Ogievskiy wrote: Add helper that returns both status and output, to be used in the following commit Signed-off-by: Vladimir Sementsov-Ogievskiy --- tests/qemu-iotests/iotests.py | 4 1 file changed, 4 insertions(+) diff --git a/tests/qemu-iotests/i

Re: [PATCH v4 17/18] qapi: backup: add immutable-source parameter

2022-02-24 Thread Hanna Reitz
On 16.02.22 20:46, Vladimir Sementsov-Ogievskiy wrote: We are on the way to implement internal-backup with fleecing scheme, which includes backup job copying from fleecing block driver node (which is target of copy-before-write filter) to final target of backup. This job doesn't need own filter,

Re: [PATCH v4 04/18] block/copy-before-write: add bitmap open parameter

2022-02-24 Thread Vladimir Sementsov-Ogievskiy
24.02.2022 15:07, Hanna Reitz wrote: On 16.02.22 20:46, Vladimir Sementsov-Ogievskiy wrote: This brings "incremental" mode to copy-before-write filter: user can specify bitmap so that filter will copy only "dirty" areas. Signed-off-by: Vladimir Sementsov-Ogievskiy ---   qapi/block-core.json   

Re: [PATCH v4 12/18] block: copy-before-write: realize snapshot-access API

2022-02-24 Thread Vladimir Sementsov-Ogievskiy
24.02.2022 15:46, Hanna Reitz wrote: On 16.02.22 20:46, Vladimir Sementsov-Ogievskiy wrote: Current scheme of image fleecing looks like this: [guest]    [NBD export]    |  |    |root  | root    v  v

Re: [PATCH v4 14/18] iotests.py: add qemu_io_pipe_and_status()

2022-02-24 Thread Vladimir Sementsov-Ogievskiy
24.02.2022 15:52, Hanna Reitz wrote: On 16.02.22 20:46, Vladimir Sementsov-Ogievskiy wrote: Add helper that returns both status and output, to be used in the following commit Signed-off-by: Vladimir Sementsov-Ogievskiy ---   tests/qemu-iotests/iotests.py | 4   1 file changed, 4 insertions

Re: [PATCH v3 0/4] Improve integration of iotests in the meson test harness

2022-02-24 Thread Paolo Bonzini
On 2/23/22 10:38, Thomas Huth wrote: This way, we can now finally get the output of failed tests on the console again (unless you're running meson test in verbose mode, where meson only puts this to the log file - for incomprehensible reasons), It's a bug (introduced by yours truly, which perha

Re: [PATCH v4 15/18] iotests/image-fleecing: add test case with bitmap

2022-02-24 Thread Vladimir Sementsov-Ogievskiy
24.02.2022 15:58, Hanna Reitz wrote: On 16.02.22 20:46, Vladimir Sementsov-Ogievskiy wrote: Note that reads zero areas (not dirty in the bitmap) fails, that's correct. Signed-off-by: Vladimir Sementsov-Ogievskiy ---   tests/qemu-iotests/tests/image-fleecing | 32 ++--   tests/qemu-iotes

Re: [PATCH v2 1/2] block/curl.c: Set error message string if curl_init_state() fails

2022-02-24 Thread Hanna Reitz
On 22.02.22 16:23, Peter Maydell wrote: In curl_open(), the 'out' label assumes that the state->errmsg string has been set (either by curl_easy_perform() or by manually copying a string into it); however if curl_init_state() fails we will jump to that label without setting the string. Add the mi

Re: [PATCH v2 2/2] block/curl.c: Check error return from curl_easy_setopt()

2022-02-24 Thread Hanna Reitz
On 22.02.22 16:23, Peter Maydell wrote: Coverity points out that we aren't checking the return value from curl_easy_setopt() for any of the calls to it we make in block/curl.c. Some of these options are documented as always succeeding (e.g. CURLOPT_VERBOSE) but others have documented failure cas

Re: [PATCH v2 0/2] block/curl: check error return from curl_easy_setopt()

2022-02-24 Thread Hanna Reitz
On 22.02.22 16:23, Peter Maydell wrote: Coverity points out that we aren't checking the return value from curl_easy_setopt() for any of the calls to it we make in block/curl.c. Tested with 'make check' and with some basic smoke test command lines suggested by Dan: qemu-img info https://cloud

Re: [PATCH v4 17/18] qapi: backup: add immutable-source parameter

2022-02-24 Thread Vladimir Sementsov-Ogievskiy
24.02.2022 16:05, Hanna Reitz wrote: On 16.02.22 20:46, Vladimir Sementsov-Ogievskiy wrote: We are on the way to implement internal-backup with fleecing scheme, which includes backup job copying from fleecing block driver node (which is target of copy-before-write filter) to final target of back

Re: [PATCH v2 2/2] block/curl.c: Check error return from curl_easy_setopt()

2022-02-24 Thread Peter Maydell
On Thu, 24 Feb 2022 at 14:11, Hanna Reitz wrote: > > On 22.02.22 16:23, Peter Maydell wrote: > > Coverity points out that we aren't checking the return value > > from curl_easy_setopt() for any of the calls to it we make > > in block/curl.c. > > > > Some of these options are documented as always s

Re: [PATCH] block: fix preallocate filter: don't do unaligned preallocate requests

2022-02-24 Thread Hanna Reitz
On 15.02.22 13:16, Vladimir Sementsov-Ogievskiy wrote: There is a bug in handling BDRV_REQ_NO_WAIT flag: we still may wait in wait_serialising_requests() if request is unaligned. And this is possible for the only user of this flag (preallocate filter) if underlying file is unaligned to its reques

Re: [PATCH] tests/qemu-iotests/040: Skip TestCommitWithFilters without 'throttle'

2022-02-24 Thread Hanna Reitz
On 23.02.22 13:31, Thomas Huth wrote: iotest 040 already has some checks for the availability of the 'throttle' driver, but some new code has been added in the course of time that depends on 'throttle' but does not check for its availability. Add a check to the TestCommitWithFilters class so that

Re: [PATCH v5 08/20] jobs: protect jobs with job_lock/unlock

2022-02-24 Thread Stefan Hajnoczi
On Thu, Feb 24, 2022 at 01:45:48PM +0100, Emanuele Giuseppe Esposito wrote: > > > On 17/02/2022 15:48, Stefan Hajnoczi wrote: > > On Tue, Feb 08, 2022 at 09:35:01AM -0500, Emanuele Giuseppe Esposito wrote: > >> diff --git a/block/replication.c b/block/replication.c > >> index 55c8f894aa..a03b2872

Re: [PATCH v5 08/20] jobs: protect jobs with job_lock/unlock

2022-02-24 Thread Emanuele Giuseppe Esposito
On 24/02/2022 17:48, Stefan Hajnoczi wrote: > On Thu, Feb 24, 2022 at 01:45:48PM +0100, Emanuele Giuseppe Esposito wrote: >> >> >> On 17/02/2022 15:48, Stefan Hajnoczi wrote: >>> On Tue, Feb 08, 2022 at 09:35:01AM -0500, Emanuele Giuseppe Esposito wrote: diff --git a/block/replication.c b/b

Re: [PATCH] tests/qemu-iotests/testrunner: Quote "case not run" lines in TAP mode

2022-02-24 Thread Hanna Reitz
On 23.02.22 13:43, Thomas Huth wrote: In TAP mode, the stdout is reserved for the TAP protocol, so we have to make sure to mark other lines with a comment '#' character at the beginning to avoid that the TAP parser at the other end gets confused. To test this condition, run "configure" for examp

[PATCH 0/2] blockdev-add transaction

2022-02-24 Thread Vladimir Sementsov-Ogievskiy
Hi all! If we want to do incremental backups with help of copy-before-write filter bitmap parameter introduced in my in-flight series "[PATCH v4 00/18] Make image fleecing more usable", we actually need to create filter, insert it into graph and do some operations with bitmaps in one transaction.

[PATCH 1/2] block: transaction support for blockdev-add

2022-02-24 Thread Vladimir Sementsov-Ogievskiy
Simply do blockdev_add() in .prepare() and bdrv_unref() in .abort() and that's it. Signed-off-by: Vladimir Sementsov-Ogievskiy --- qapi/transaction.json | 11 ++ blockdev.c| 80 +-- 2 files changed, 66 insertions(+), 25 deletions(-) diff -

[PATCH 2/2] iotests: add blockdev-add-transaction

2022-02-24 Thread Vladimir Sementsov-Ogievskiy
Add a test for transaction support of blockdev-add. Test is format-agnostic, so limit it to qcow2 to avoid extra test runs. Signed-off-by: Vladimir Sementsov-Ogievskiy --- .../tests/blockdev-add-transaction| 52 +++ .../tests/blockdev-add-transaction.out| 6

[PATCH v4] tests/qtest: add qtests for npcm7xx sdhci

2022-02-24 Thread Hao Wu
From: Shengtan Mao Reviewed-by: Hao Wu Reviewed-by: Chris Rauer Signed-off-by: Shengtan Mao Signed-off-by: Patrick Venture Signed-off-by: Hao Wu --- v4: * use strncmp to compare fixed length strings v3: * fixup compilation from missing macro value v2: * update copyright year * check resu

Re: [PATCH] aio-posix: fix spurious ->poll_ready() callbacks in main loop

2022-02-24 Thread Jason Wang
在 2022/2/23 下午11:57, Stefan Hajnoczi 写道: When ->poll() succeeds the AioHandler is placed on the ready list with revents set to the magic value 0. This magic value causes aio_dispatch_handler() to invoke ->poll_ready() instead of ->io_read() for G_IO_IN or ->io_write() for G_IO_OUT. This magic