[PATCH v5 00/16] Make image fleecing more usable

v5: Drop immutable_source backup parameter: it's not necessary with new scheme. Other changes: 03,06,07,08,10,13 : add Hanna's r-bs 04: add (Since 7.0) 11: s/filter/block driver/, also add note about snapshot size 12: fix text and s/bs/child->bs/kk update iotest 257 output 14: use qemu_io_wrap

[PATCH v5 01/16] block/block-copy: move copy_bitmap initialization to block_copy_state_new()

We are going to complicate bitmap initialization in the further commit. And in future, backup job will be able to work without filter (when source is immutable), so we'll need same bitmap initialization in copy-before-write filter and in backup job. So, it's reasonable to do it in block-copy. Note

[PATCH v5 04/16] block/copy-before-write: add bitmap open parameter

This brings "incremental" mode to copy-before-write filter: user can specify bitmap so that filter will copy only "dirty" areas. Signed-off-by: Vladimir Sementsov-Ogievskiy --- qapi/block-core.json | 10 +++- block/copy-before-write.c | 51 ++- 2 file

[PATCH v5 11/16] block: introduce snapshot-access block driver

The new block driver simply utilizes snapshot-access API of underlying block node. In further patches we want to use it like this: [guest] [NBD export] || | root | root v file v [copy-before-write]<

[PATCH v5 15/16] iotests/image-fleecing: add test case with bitmap

Note that reads zero areas (not dirty in the bitmap) fails, that's correct. Signed-off-by: Vladimir Sementsov-Ogievskiy --- tests/qemu-iotests/tests/image-fleecing | 32 ++-- tests/qemu-iotests/tests/image-fleecing.out | 84 + 2 files changed, 108 insertions(+), 8 del

[PATCH v5 05/16] block/block-copy: add block_copy_reset()

Split block_copy_reset() out of block_copy_reset_unallocated() to be used separately later. Signed-off-by: Vladimir Sementsov-Ogievskiy Reviewed-by: Hanna Reitz --- include/block/block-copy.h | 1 + block/block-copy.c | 21 + 2 files changed, 14 insertions(+), 8 del

[PATCH v5 02/16] block/dirty-bitmap: bdrv_merge_dirty_bitmap(): add return value

That simplifies handling failure in existing code and in further new usage of bdrv_merge_dirty_bitmap(). Signed-off-by: Vladimir Sementsov-Ogievskiy Reviewed-by: Hanna Reitz --- include/block/dirty-bitmap.h| 2 +- block/dirty-bitmap.c| 9 +++-- block/monitor/bitmap-qmp-cmds.

[PATCH v5 08/16] block/dirty-bitmap: introduce bdrv_dirty_bitmap_status()

Add a convenient function similar with bdrv_block_status() to get status of dirty bitmap. Signed-off-by: Vladimir Sementsov-Ogievskiy Reviewed-by: Hanna Reitz --- include/block/dirty-bitmap.h | 2 ++ include/qemu/hbitmap.h | 12 block/dirty-bitmap.c | 6 ++ util

[PATCH v5 09/16] block/reqlist: add reqlist_wait_all()

Add function to wait for all intersecting requests. To be used in the further commit. Signed-off-by: Vladimir Sementsov-Ogievskiy Reviewed-by: Nikita Lapshin Reviewed-by: Hanna Reitz --- include/block/reqlist.h | 8 block/reqlist.c | 8 2 files changed, 16 insertions(

[PATCH v5 03/16] block/block-copy: block_copy_state_new(): add bitmap parameter

This will be used in the following commit to bring "incremental" mode to copy-before-write filter. Signed-off-by: Vladimir Sementsov-Ogievskiy Reviewed-by: Hanna Reitz --- include/block/block-copy.h | 1 + block/block-copy.c | 14 +- block/copy-before-write.c | 2 +- 3 fi

[PATCH v5 06/16] block: intoduce reqlist

Split intersecting-requests functionality out of block-copy to be reused in copy-before-write filter. Note: while being here, fix tiny typo in MAINTAINERS. Signed-off-by: Vladimir Sementsov-Ogievskiy Reviewed-by: Hanna Reitz --- include/block/reqlist.h | 67 +++ block/bloc

[PATCH v5 07/16] block/reqlist: reqlist_find_conflict(): use ranges_overlap()

Let's reuse convenient helper. Signed-off-by: Vladimir Sementsov-Ogievskiy Reviewed-by: Hanna Reitz --- block/reqlist.c | 3 ++- 1 file changed, 2 insertions(+), 1 deletion(-) diff --git a/block/reqlist.c b/block/reqlist.c index 5e320ba649..09fecbd48c 100644 --- a/block/reqlist.c +++ b/block/r

[PATCH v5 10/16] block/io: introduce block driver snapshot-access API

Add new block driver handlers and corresponding generic wrappers. It will be used to allow copy-before-write filter to provide reach fleecing interface in further commit. In future this approach may be used to allow reading qcow2 internal snapshots, for example to export them through NBD. Signed-

[PATCH v5 14/16] iotests.py: add qemu_io_pipe_and_status()

Add helper that returns both status and output, to be used in the following commit Signed-off-by: Vladimir Sementsov-Ogievskiy --- tests/qemu-iotests/iotests.py | 3 +++ 1 file changed, 3 insertions(+) diff --git a/tests/qemu-iotests/iotests.py b/tests/qemu-iotests/iotests.py index 6ba65eb1ff..

[PATCH v5 16/16] iotests/image-fleecing: test push backup with fleecing

Signed-off-by: Vladimir Sementsov-Ogievskiy --- tests/qemu-iotests/tests/image-fleecing | 120 ++-- tests/qemu-iotests/tests/image-fleecing.out | 63 ++ 2 files changed, 151 insertions(+), 32 deletions(-) diff --git a/tests/qemu-iotests/tests/image-fleecing b/tests/

[PATCH v5 12/16] block: copy-before-write: realize snapshot-access API

Current scheme of image fleecing looks like this: [guest][NBD export] | | |root | root v v [copy-before-write] -> [temp.qcow2] | target | |file

[PATCH v5 13/16] iotests/image-fleecing: add test-case for fleecing format node

Signed-off-by: Vladimir Sementsov-Ogievskiy Reviewed-by: Hanna Reitz --- tests/qemu-iotests/tests/image-fleecing | 64 - tests/qemu-iotests/tests/image-fleecing.out | 76 - 2 files changed, 120 insertions(+), 20 deletions(-) diff --git a/tests/qemu-iotest

Re: [PATCH v2 0/3] block: Make bdrv_refresh_limits() non-recursive

On Wed, Feb 16, 2022 at 11:53:52AM +0100, Hanna Reitz wrote: > Hi, > > v1 with detailed reasoning: > https://lists.nongnu.org/archive/html/qemu-block/2022-02/msg00508.html > > This series makes bdrv_refresh_limits() non-recursive so that it is > sufficient for callers to ensure that the node on w

Re: [PATCH 1/3] util & iothread: Introduce event-loop abstract class

Hi Stefan, thanks for the review. On Thu, 2022-02-24 at 09:48 +, Stefan Hajnoczi wrote: > On Mon, Feb 21, 2022 at 06:08:43PM +0100, Nicolas Saenz Julienne wrote: > > diff --git a/qom/meson.build b/qom/meson.build > > index 062a3789d8..c20e5dd1cb 100644 > > --- a/qom/meson.build > > +++ b/qom/m

Re: [PATCH 2/3] util/main-loop: Introduce the main loop into QOM

On Thu, 2022-02-24 at 10:01 +, Stefan Hajnoczi wrote: > On Mon, Feb 21, 2022 at 06:08:44PM +0100, Nicolas Saenz Julienne wrote: > > diff --git a/include/qemu/main-loop.h b/include/qemu/main-loop.h > > index 8dbc6fcb89..fea5a3e9d4 100644 > > --- a/include/qemu/main-loop.h > > +++ b/include/qemu/

Re: [PATCH 3/3] util/event-loop: Introduce options to set the thread pool size

On Thu, 2022-02-24 at 10:40 +, Stefan Hajnoczi wrote: > On Mon, Feb 21, 2022 at 06:08:45PM +0100, Nicolas Saenz Julienne wrote: > > The thread pool regulates itself: when idle, it kills threads until > > empty, when in demand, it creates new threads until full. This behaviour > > doesn't play w

Re: [PATCH 1/9] hw/usb/redirect.c: Stop using qemu_oom_check()

On 26/2/22 19:07, Peter Maydell wrote: qemu_oom_check() is a function which essentially says "if you pass me a NULL pointer then print a message then abort()". On POSIX systems the message includes strerror(errno); on Windows it includes the GetLastError() error value printed as an integer. Oth

Re: [PATCH 7/9] util: Use meson checks for valloc() and memalign() presence

On 26/2/22 19:07, Peter Maydell wrote: Instead of assuming that all CONFIG_BSD have valloc() and anything else is memalign(), explicitly check for those functions in meson.build and use the "is the function present" define. Tests for specific functionality are better than which-OS checks; this a

Re: [PATCH 2/9] util: Make qemu_oom_check() a static function

On 26/2/22 19:07, Peter Maydell wrote: The qemu_oom_check() function, which we define in both oslib-posix.c and oslib-win32.c, is now used only locally in that file; make it static. Signed-off-by: Peter Maydell --- include/qemu-common.h | 2 -- util/oslib-posix.c| 2 +- util/oslib-win32

Re: [PATCH 9/9] osdep: Move memalign-related functions to their own header

On 26/2/22 19:07, Peter Maydell wrote: Move the various memalign-related functions out of osdep.h and into their own header, which we include only where they are used. While we're doing this, add some brief documentation comments. Signed-off-by: Peter Maydell --- include/qemu/memalign.h

Re: [PATCH v8 5/5] multifd: Implement zero copy write in multifd migration (multifd-zero-copy)

On Fri, Feb 18, 2022 at 05:57:13PM +0100, Juan Quintela wrote: > I did a change on: > > commit d48c3a044537689866fe44e65d24c7d39a68868a > Author: Juan Quintela > Date: Fri Nov 19 15:35:58 2021 +0100 > > multifd: Use a single writev on the send side > > Until now, we wrote the pack