Re: [PATCH v2 00/11] qcow2: make subclusters discardable

2024-06-03 Thread Andrey Drobyshev
On 5/13/24 9:31 AM, Andrey Drobyshev wrote: > v1: https://lists.nongnu.org/archive/html/qemu-devel/2023-10/msg07223.html > > Andrey Drobyshev (11): > qcow2: make function update_refcount_discard() global > qcow2: simplify L2 entries accounting for discard-no-unref >

Re: [PATCH v2 00/11] qcow2: make subclusters discardable

2024-06-10 Thread Andrey Drobyshev
On 6/3/24 12:19 PM, Andrey Drobyshev wrote: > On 5/13/24 9:31 AM, Andrey Drobyshev wrote: >> v1: https://lists.nongnu.org/archive/html/qemu-devel/2023-10/msg07223.html >> >> Andrey Drobyshev (11): >> qcow2: make function update_refcount_discard() global >> qcow

Re: [PATCH v2 00/11] qcow2: make subclusters discardable

2024-06-17 Thread Andrey Drobyshev
On 6/10/24 11:53 AM, Andrey Drobyshev wrote: > On 6/3/24 12:19 PM, Andrey Drobyshev wrote: >> On 5/13/24 9:31 AM, Andrey Drobyshev wrote: >>> v1: https://lists.nongnu.org/archive/html/qemu-devel/2023-10/msg07223.html >>> >>> Andrey Drobyshev (11): >>>

Re: [PATCH v2 00/11] qcow2: make subclusters discardable

2024-06-24 Thread Andrey Drobyshev
On 6/17/24 9:39 AM, Andrey Drobyshev wrote: > On 6/10/24 11:53 AM, Andrey Drobyshev wrote: >> On 6/3/24 12:19 PM, Andrey Drobyshev wrote: >>> On 5/13/24 9:31 AM, Andrey Drobyshev wrote: >>>> v1: https://lists.nongnu.org/archive/html/qemu-devel/2023-10/msg07223.html

Re: [PATCH v2 00/11] qcow2: make subclusters discardable

2024-07-08 Thread Andrey Drobyshev
On 6/24/24 10:43 AM, Andrey Drobyshev wrote: > On 6/17/24 9:39 AM, Andrey Drobyshev wrote: >> On 6/10/24 11:53 AM, Andrey Drobyshev wrote: >>> On 6/3/24 12:19 PM, Andrey Drobyshev wrote: >>>> On 5/13/24 9:31 AM, Andrey Drobyshev wrote: >>>>> v1: htt

[PATCH 0/2] Fix data corruption within preallocation

2024-07-11 Thread Andrey Drobyshev
7;re other operations which are run outside of the coroutine context and which also modify the file_end field, e.g. preallocate_set_perm() and preallocate_child_perm(). Comments on how to protect the field properly regardless of the context are welcome. Andrey Drobyshev (1): iotests/298: add testcase

[PATCH 2/2] iotests/298: add testcase for async writes with preallocation filter

2024-07-11 Thread Andrey Drobyshev
esult into a race within the preallocation filter which would zeroize some clusters after actually writing data to them. Note: the test doesn't fail in 100% cases as there's a race involved, but the failures are pretty consistent so it should be good enough for detecting the problem. S

[PATCH 1/2] block: zero data data corruption using prealloc-filter

2024-07-11 Thread Andrey Drobyshev
only once it is detected that the preallocation is really required. This is not a frequent case due to the preallocation nature thus the patch should not have performance impact. Originally-by: Denis V. Lunev Co-authored-by: Andrey Drobyshev Signed-off-by: Denis V. Lunev Signed-off-by: And

[PATCH v2 1/2] block: zero data data corruption using prealloc-filter

2024-07-12 Thread Andrey Drobyshev
this area and they will not proceed thanks to overlapping requests mechanics. If preallocation will fail, we will just switch to the normal expand-by-write behavior and that is not a problem except performance. Signed-off-by: Denis V. Lunev Tested-by: Andrey Drobyshev --- block/preallocate.c |

[PATCH v2 2/2] iotests/298: add testcase for async writes with preallocation filter

2024-07-12 Thread Andrey Drobyshev
esult into a race within the preallocation filter which would zeroize some clusters after actually writing data to them. Note: the test doesn't fail in 100% cases as there's a race involved, but the failures are pretty consistent so it should be good enough for detecting the problem. S

[PATCH v2 0/2] Fix data corruption within preallocation

2024-07-12 Thread Andrey Drobyshev
the operations performed in other coroutines (e.g. truncate) and outside of the coroutine context (preallocate_set_perm() / preallocate_child_perm()). So the comments on this are most welcome. v1: https://lists.nongnu.org/archive/html/qemu-block/2024-07/msg00384.html Andrey Drobyshev (1): iotests/

Re: [PATCH v2 00/11] qcow2: make subclusters discardable

2024-07-15 Thread Andrey Drobyshev
On 7/8/24 10:06 AM, Andrey Drobyshev wrote: > On 6/24/24 10:43 AM, Andrey Drobyshev wrote: >> On 6/17/24 9:39 AM, Andrey Drobyshev wrote: >>> On 6/10/24 11:53 AM, Andrey Drobyshev wrote: >>>> On 6/3/24 12:19 PM, Andrey Drobyshev wrote: >>>>> On 5/13/

[PATCH v2 3/2] scripts: add filev2p.py script for mapping virtual file offsets mapping

2024-07-15 Thread Andrey Drobyshev
;t work with partitions built on top of LVM. Partitions on LVM would require another level of mapping. Signed-off-by: Andrey Drobyshev --- scripts/filev2p.py | 311 + 1 file changed, 311 insertions(+) create mode 100755 scripts/filev2p.py diff --gi

Re: [PATCH v2 1/2] block: zero data data corruption using prealloc-filter

2024-07-16 Thread Andrey Drobyshev
On 7/16/24 4:32 PM, Denis V. Lunev wrote: > On 7/12/24 13:55, Vladimir Sementsov-Ogievskiy wrote: >> On 12.07.24 12:46, Andrey Drobyshev wrote: >>> From: "Denis V. Lunev" >>> >>> We have observed that some clusters in the QCOW2 files are zeroed >

[PATCH v3 1/3] block: zero data data corruption using prealloc-filter

2024-07-16 Thread Andrey Drobyshev
this area and they will not proceed thanks to overlapping requests mechanics. If preallocation will fail, we will just switch to the normal expand-by-write behavior and that is not a problem except performance. Signed-off-by: Denis V. Lunev Tested-by: Andrey Drobyshev --- block/preallocate.c |

[PATCH v3 2/3] iotests/298: add testcase for async writes with preallocation filter

2024-07-16 Thread Andrey Drobyshev
esult into a race within the preallocation filter which would zeroize some clusters after actually writing data to them. Note: the test doesn't fail in 100% cases as there's a race involved, but the failures are pretty consistent so it should be good enough for detecting the problem. S

[PATCH v3 3/3] scripts: add filev2p.py script for mapping virtual file offsets mapping

2024-07-16 Thread Andrey Drobyshev
;t work with partitions built on top of LVM. Partitions on LVM would require another level of mapping. Signed-off-by: Andrey Drobyshev --- scripts/filev2p.py | 311 + 1 file changed, 311 insertions(+) create mode 100755 scripts/filev2p.py diff --gi

[PATCH v3 0/3] Fix data corruption within preallocation

2024-07-16 Thread Andrey Drobyshev
ets. The script was used to initially track down the data corruption problem, so it's included here. v2: https://lists.nongnu.org/archive/html/qemu-block/2024-07/msg00413.html Andrey Drobyshev (2): iotests/298: add testcase for async writes with preallocation filter scri

Re: [PULL 06/27] iotests/{024, 271}: add testcases for qemu-img rebase

2024-07-30 Thread Andrey Drobyshev
On 7/22/24 10:18 AM, Thomas Huth wrote: > [Вы нечасто получаете письма от th...@redhat.com. Узнайте, почему это > важно, по адресу https://aka.ms/LearnAboutSenderIdentification ] > > On 31/10/2023 19.58, Kevin Wolf wrote: >> From: Andrey Drobyshev >> >> As the pre

[PATCH] iotests/024: exclude 'backing file format' field from the output

2024-07-30 Thread Andrey Drobyshev
m the output. This is a fixup for the commit f93e65ee51 ("iotests/{024, 271}: add testcases for qemu-img rebase"). Found-by: Thomas Huth Signed-off-by: Andrey Drobyshev --- tests/qemu-iotests/024 | 2 +- tests/qemu-iotests/024.out | 1 - 2 files changed, 1 insertion(+), 2 deletion

Re: [PATCH v3 2/3] iotests/298: add testcase for async writes with preallocation filter

2024-08-05 Thread Andrey Drobyshev
On 8/5/24 3:04 PM, Kevin Wolf wrote: > Am 16.07.2024 um 16:41 hat Andrey Drobyshev geschrieben: >> The testcase simply creates a 64G image with 1M clusters, generates a list >> of 1M aligned offsets and feeds aio_write commands with those offsets to >> qemu-io run with &#

Re: [PATCH v3 3/3] scripts: add filev2p.py script for mapping virtual file offsets mapping

2024-08-05 Thread Andrey Drobyshev
On 8/5/24 3:29 PM, Kevin Wolf wrote: > Am 16.07.2024 um 16:41 hat Andrey Drobyshev geschrieben: >> The script is basically a wrapper around "filefrag" utility. This might >> be used to map virtual offsets within the file to the underlying block >> device offsets.

[PATCH v3 06/12] iotests/290: add test case to check 'discard-no-unref' option behavior

2024-09-13 Thread Andrey Drobyshev
) and that with the option enabled cluster is still marked as allocated in "qemu-img map" output. We also check that the option doesn't work with qcow2 v2 images. Signed-off-by: Andrey Drobyshev Reviewed-by: Alexander Ivanov Reviewed-by: Alberto Garcia --- tests/

[PATCH v3 01/12] qcow2: make function update_refcount_discard() global

2024-09-13 Thread Andrey Drobyshev
We are going to need it for discarding separate subclusters. The function itself doesn't do anything with the refcount tables, it simply adds a discard request to the queue, so rename it to qcow2_queue_discard(). Signed-off-by: Andrey Drobyshev Reviewed-by: Alexander Ivanov Reviewed-by:

[PATCH v3 08/12] qcow2: zeroize the entire cluster when there're no non-zero subclusters

2024-09-13 Thread Andrey Drobyshev
When zeroizing the last non-zero subclusters within single cluster, it makes sense to go zeroize the entire cluster and go down zero_in_l2_slice() path right away. That way we'd also update the corresponding refcount table. Signed-off-by: Andrey Drobyshev Reviewed-by: Alexander Ivanov Rev

[PATCH v3 00/12] qcow2: make subclusters discardable

2024-09-13 Thread Andrey Drobyshev
v2: https://lists.nongnu.org/archive/html/qemu-devel/2024-05/msg02396.html v2 -> v3: * Added patch 12/12 "qcow2: add discard-subclusters option" which makes subcluster-based discards optional; * Added a bunch of R-b's. Andrey Drobyshev (12): qcow2: make function updat

[PATCH v3 03/12] qcow2: put discard requests in the common queue when discard-no-unref enabled

2024-09-13 Thread Andrey Drobyshev
check for the presence of external data files for us and redirect request to underlying data_file. Here we want to do the same but avoid refcount updates, thus we perform the same checks. Suggested-by: Hanna Czenczek Signed-off-by: Andrey Drobyshev Reviewed-by: Alexander Ivanov --- block/

[PATCH v3 05/12] iotests/common.rc: add disk_usage function

2024-09-13 Thread Andrey Drobyshev
Move the definition from iotests/250 to common.rc. This is used to detect real disk usage of sparse files. In particular, we want to use it for checking subclusters-based discards. Signed-off-by: Andrey Drobyshev Reviewed-by: Alexander Ivanov Reviewed-by: Alberto Garcia --- tests/qemu

[PATCH v3 11/12] iotests/271: add test cases for subcluster-based discard/unmap

2024-09-13 Thread Andrey Drobyshev
make all discard/unmap operations enable trace point 'file_do_fallocate' so that actual fallocate() calls are visible. Signed-off-by: Andrey Drobyshev Reviewed-by: Alexander Ivanov --- tests/qemu-iotests/271 | 70 +- tests/qemu-iotests/27

[PATCH v3 09/12] qcow2: make subclusters discardable

2024-09-13 Thread Andrey Drobyshev
with subclusters enabled. Also rename qcow2_cluster_discard() -> qcow2_subcluster_discard() to reflect the change. Signed-off-by: Andrey Drobyshev Reviewed-by: Alexander Ivanov --- block/qcow2-cluster.c | 106 + block/qcow2-snapshot.c | 6 +-- block

[PATCH v3 04/12] block/file-posix: add trace event for fallocate() calls

2024-09-13 Thread Andrey Drobyshev
This would ease debugging of write zeroes and discard operations. Signed-off-by: Andrey Drobyshev Reviewed-by: Alexander Ivanov Reviewed-by: Alberto Garcia --- block/file-posix.c | 1 + block/trace-events | 1 + 2 files changed, 2 insertions(+) diff --git a/block/file-posix.c b/block/file

[PATCH v3 12/12] qcow2: add discard-subclusters option

2024-09-13 Thread Andrey Drobyshev
eally test subcluster based discards. Signed-off-by: Andrey Drobyshev --- block/qcow2.c | 21 - block/qcow2.h | 2 ++ tests/qemu-iotests/271 | 10 ++ 3 files changed, 28 insertions(+), 5 deletions(-) diff --git a/block/qcow2.c b/block/qcow2.c index

[PATCH v3 02/12] qcow2: simplify L2 entries accounting for discard-no-unref

2024-09-13 Thread Andrey Drobyshev
atter whether we unmap or zeroize the cluster. For that OR'ing with the old entry is enough. This patch doesn't change the logic and is pure refactoring. Signed-off-by: Andrey Drobyshev Reviewed-by: Alexander Ivanov Reviewed-by: Alberto Garcia --- block/qcow2

[PATCH v3 07/12] qcow2: add get_sc_range_info() helper for working with subcluster ranges

2024-09-13 Thread Andrey Drobyshev
ation. Also introduce struct SubClusterRangeInfo, which would contain all the needed params. Signed-off-by: Andrey Drobyshev Reviewed-by: Alexander Ivanov --- block/qcow2-cluster.c | 140 -- 1 file changed, 108 insertions(+), 32 deletions(-) diff --git a/

[PATCH v3 10/12] qcow2: zero_l2_subclusters: fall through to discard operation when requested

2024-09-13 Thread Andrey Drobyshev
quot; will lead to actual unmap. Signed-off-by: Andrey Drobyshev Reviewed-by: Alexander Ivanov --- block/qcow2-cluster.c | 26 -- 1 file changed, 20 insertions(+), 6 deletions(-) diff --git a/block/qcow2-cluster.c b/block/qcow2-cluster.c index 3c134a7e80..53e04eff93

Re: [PATCH v3 12/12] qcow2: add discard-subclusters option

2024-09-15 Thread Andrey Drobyshev
On 9/13/24 7:39 PM, Andrey Drobyshev wrote: > Introduce Qcow2 runtime boolean option "discard-subclusters". This > option influences discard alignment value (either cluster_size or > subcluster_size) and essentially makes subcluster-based discard optional. > We disable it

Re: [PATCH 1/2] qemu-img: rebase: stop when reaching EOF of old backing file

2023-05-25 Thread Andrey Drobyshev
On 5/24/23 11:30, Denis V. Lunev wrote: > On 5/23/23 18:24, Andrey Drobyshev wrote: >> In case when we're rebasing within one backing chain, and when target >> image >> is larger than old backing file, bdrv_is_allocated_above() ends up >> setting >> *pnum

Re: [PATCH v2 0/2] qemu-img: fix getting stuck in infinite loop on in-chain rebase

2023-06-01 Thread Andrey Drobyshev
On 5/25/23 21:02, Andrey Drobyshev wrote: > v1 -> v2: > > * Avoid breaking the loop just yet, as the offsets beyond the old > backing size need to be explicitly zeroed; > * Amend the commit message accordingly; > * Alter the added test case to take the las

Re: [PATCH 1/6] qemu-img: rebase: stop when reaching EOF of old backing file

2023-06-02 Thread Andrey Drobyshev
On 6/2/23 00:18, Michael Tokarev wrote: > 01.06.2023 22:28, Andrey Drobyshev via пишет: >> In case when we're rebasing within one backing chain, and when target >> image >> is larger than old backing file, bdrv_is_allocated_above() ends up >> setting >> *pnum

Re: [PATCH 1/3] block: add BDRV_BLOCK_COMPRESSED flag for bdrv_block_status()

2023-06-21 Thread Andrey Drobyshev
On 6/21/23 20:46, Denis V. Lunev wrote: > On 6/21/23 19:08, Denis V. Lunev wrote: >> On 6/7/23 17:26, Andrey Drobyshev wrote: >>> Functions qcow2_get_host_offset(), get_cluster_offset() explicitly >>> report compressed cluster types when data is compressed. However, th

Re: [PATCH 2/3] qemu-img: map: report compressed data blocks

2023-07-06 Thread Andrey Drobyshev
On 6/21/23 21:12, Denis V. Lunev wrote: > On 6/7/23 17:26, Andrey Drobyshev wrote: >> Right now "qemu-img map" reports compressed blocks as containing data >> but having no host offset.  This is not very informative.  Instead, >> let's add another boolean f

Re: [PATCH 0/6] qemu-img: rebase: add compression support

2023-07-07 Thread Andrey Drobyshev
On 6/30/23 13:54, Denis V. Lunev wrote: > On 6/1/23 21:28, Andrey Drobyshev wrote: >> This series is adding [-c | --compress] option to "qemu-img rebase" >> command, which might prove useful for saving some disk space when, for >> instance, manipulating chains of

Re: [PATCH 0/3] block: align CoR requests to subclusters

2023-07-07 Thread Andrey Drobyshev
On 6/26/23 19:08, Andrey Drobyshev wrote: > This series makes IO requests performed with copy-on-read to be aligned > to subclusters rather than clusters. It also affects mirror job requests > alignment. > > The initial reason for that change is the following crash discovered

Re: [PATCH 2/3] block/io: align requests to subcluster_size

2023-07-11 Thread Andrey Drobyshev
On 7/10/23 22:47, Eric Blake wrote: > On Mon, Jun 26, 2023 at 07:08:33PM +0300, Andrey Drobyshev via wrote: >> When target image is using subclusters, and we align the request during >> copy-on-read, it makes sense to align to subcluster_size rather than >> cluster_size. Oth

Re: [PATCH 2/7] qcow2: add get_sc_range_info() helper for working with subcluster ranges

2023-11-09 Thread Andrey Drobyshev
Hello Hanna, Sorry for the delay and thanks for your thorough and detailed review. On 10/31/23 17:53, Hanna Czenczek wrote: > On 20.10.23 23:56, Andrey Drobyshev wrote: >> This helper simply obtains the l2 table parameters of the cluster which >> contains the given subclusters ra

Re: [PATCH 6/7] iotests/common.rc: add disk_usage function

2023-11-09 Thread Andrey Drobyshev
On 11/3/23 17:20, Hanna Czenczek wrote: > On 20.10.23 23:56, Andrey Drobyshev wrote: >> Move the definition from iotests/250 to common.rc.  This is used to >> detect real disk usage of sparse files.  In particular, we want to use >> it for checking subclusters-based discards

Re: [PATCH 4/7] qcow2: make subclusters discardable

2023-11-09 Thread Andrey Drobyshev
On 10/31/23 18:32, Hanna Czenczek wrote: > On 20.10.23 23:56, Andrey Drobyshev wrote: >> This commit makes the discard operation work on the subcluster level >> rather than cluster level.  It introduces discard_l2_subclusters() >> function and makes use of it in qcow2 discard

Re: [PATCH 7/7] iotests/271: check disk usage on subcluster-based discard/unmap

2023-11-09 Thread Andrey Drobyshev
On 11/3/23 17:59, Hanna Czenczek wrote: > On 03.11.23 16:51, Hanna Czenczek wrote: >> On 20.10.23 23:56, Andrey Drobyshev wrote: > > [...] > >>> @@ -528,6 +543,14 @@ for use_backing_file in yes no; do >>>   else >>>   _make_test_img -o ex

Re: [PATCH 7/7] iotests/271: check disk usage on subcluster-based discard/unmap

2023-11-09 Thread Andrey Drobyshev
On 11/3/23 17:51, Hanna Czenczek wrote: > On 20.10.23 23:56, Andrey Drobyshev wrote: >> Add _verify_du_delta() checker which is used to check that real disk >> usage delta meets the expectations.  For now we use it for checking that >> subcluster-based discard/unmap operation

Re: [PATCH 5/7] qcow2: zero_l2_subclusters: fall through to discard operation when requested

2023-11-10 Thread Andrey Drobyshev
On 11/3/23 17:19, Hanna Czenczek wrote: > On 20.10.23 23:56, Andrey Drobyshev wrote: >> When zeroizing subclusters within single cluster, detect usage of the >> BDRV_REQ_MAY_UNMAP flag and fall through to the subcluster-based discard >> operation, much like it's d

Re: [PATCH 4/7] qcow2: make subclusters discardable

2023-11-10 Thread Andrey Drobyshev
On 10/31/23 18:33, Hanna Czenczek wrote: > (Sorry, opened another reply window, forgot I already had one open...) > > On 20.10.23 23:56, Andrey Drobyshev wrote: >> This commit makes the discard operation work on the subcluster level >> rather than cluster

[PATCH] iotests: fix default MT detection

2023-11-22 Thread Andrey Drobyshev
default) (deprecated)". To fix potential issues here, let's relax that requirement and detect the mere presence of " (default)" line instead. Signed-off-by: Andrey Drobyshev --- tests/qemu-iotests/testenv.py | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/

[PATCH] block: allocate aligned write buffer for 'truncate -m full'

2023-12-11 Thread Andrey Drobyshev
#x27;s just allocate the buffer using qemu_blockalign0() instead. Signed-off-by: Andrey Drobyshev --- block/file-posix.c | 4 ++-- 1 file changed, 2 insertions(+), 2 deletions(-) diff --git a/block/file-posix.c b/block/file-posix.c index b862406c71..cee8de510b 100644 --- a/block/file-posix.c +++

[PATCH] iotests: don't run tests requiring cached writes in '-nocache' mode

2023-12-11 Thread Andrey Drobyshev
ment * 308, file-io-error: use fuse exports. Though fuse does have 'direct-io' mode (see https://docs.kernel.org/filesystems/fuse-io.html) we aren't using it yet, thus getting errors like: qemu-io: can't open device /path/to/t.qcow2.fuse: Could not open '/path/to

Re: [PATCH] block: allocate aligned write buffer for 'truncate -m full'

2024-01-11 Thread Andrey Drobyshev
On 12/11/23 13:27, Denis V. Lunev wrote: > On 12/11/23 11:55, Andrey Drobyshev wrote: >> In case we're truncating an image opened with O_DIRECT, we might get >> -EINVAL on write with unaligned buffer.  In particular, when running >> iotests/298 with '-nocache&#x

Re: [PATCH] iotests: don't run tests requiring cached writes in '-nocache' mode

2024-01-11 Thread Andrey Drobyshev
On 12/11/23 15:32, Andrey Drobyshev wrote: > There're tests whose logic implies running without O_DIRECT set, > otherwise they fail when running iotests in '-nocache' mode. For these > tests let's add _require_no_o_direct() helper which can be put in the > preabml

[PATCH] iotests/277: Use iotests.sock_dir for socket creation

2024-01-24 Thread Andrey Drobyshev
ff-by: Andrey Drobyshev --- tests/qemu-iotests/277 | 3 ++- 1 file changed, 2 insertions(+), 1 deletion(-) diff --git a/tests/qemu-iotests/277 b/tests/qemu-iotests/277 index 24833e7eb6..4224202ac2 100755 --- a/tests/qemu-iotests/277 +++ b/tests/qemu-iotests/277 @@ -27,7 +27,8 @@ from iotests import file

[PATCH] iotests/264: Use iotests.sock_dir for socket creation

2024-01-25 Thread Andrey Drobyshev
ff-by: Andrey Drobyshev --- tests/qemu-iotests/264 | 3 ++- 1 file changed, 2 insertions(+), 1 deletion(-) diff --git a/tests/qemu-iotests/264 b/tests/qemu-iotests/264 index c532ccd809..c6ba2754e2 100755 --- a/tests/qemu-iotests/264 +++ b/tests/qemu-iotests/264 @@ -25,7 +25,8 @@ import os import io

Re: [PATCH] iotests/277: Use iotests.sock_dir for socket creation

2024-01-25 Thread Andrey Drobyshev
On 1/24/24 19:59, Denis V. Lunev wrote: > On 1/24/24 18:43, Eric Blake wrote: >> On Wed, Jan 24, 2024 at 06:22:57PM +0200, Andrey Drobyshev wrote: >>> If socket path is too long (longer than 108 bytes), socket can't be >>> opened.  This might lead to failure w

Re: [PATCH] iotests: don't run tests requiring cached writes in '-nocache' mode

2024-01-25 Thread Andrey Drobyshev
On 1/11/24 14:53, Andrey Drobyshev wrote: > On 12/11/23 15:32, Andrey Drobyshev wrote: >> There're tests whose logic implies running without O_DIRECT set, >> otherwise they fail when running iotests in '-nocache' mode. For these >> tests let's add _require

Re: [PATCH] block: allocate aligned write buffer for 'truncate -m full'

2024-01-25 Thread Andrey Drobyshev
On 1/11/24 14:53, Andrey Drobyshev wrote: > On 12/11/23 13:27, Denis V. Lunev wrote: >> On 12/11/23 11:55, Andrey Drobyshev wrote: >>> In case we're truncating an image opened with O_DIRECT, we might get >>> -EINVAL on write with unaligned buffer.  In particular, w

Re: [PATCH] iotests: don't run tests requiring cached writes in '-nocache' mode

2024-01-26 Thread Andrey Drobyshev
On 1/26/24 12:24, Kevin Wolf wrote: > Am 11.12.2023 um 14:32 hat Andrey Drobyshev geschrieben: >> There're tests whose logic implies running without O_DIRECT set, >> otherwise they fail when running iotests in '-nocache' mode. For these >> tests let's ad

Re: [PATCH] block: allocate aligned write buffer for 'truncate -m full'

2024-02-08 Thread Andrey Drobyshev
On 1/25/24 18:46, Vladimir Sementsov-Ogievskiy wrote: > On 11.12.23 13:55, Andrey Drobyshev wrote: >> In case we're truncating an image opened with O_DIRECT, we might get >> -EINVAL on write with unaligned buffer.  In particular, when running >> iotests/298 with '

Re: [PATCH 4/7] qcow2: make subclusters discardable

2024-04-16 Thread Andrey Drobyshev
On 10/27/23 14:10, Jean-Louis Dupond wrote: > [...] > > I've checked all the code paths, and as far as I see it nowhere breaks > the discard_no_unref option. > It's important that we don't introduce new code paths that can make > holes in the qcow2 image when this option is enabled :) > > If you

[BUG, RFC] Base node is in RW after making external snapshot

2024-04-24 Thread Andrey Drobyshev
Hi everyone, When making an external snapshot, we end up in a situation when 2 block graph nodes related to the same image file (format and storage nodes) have different RO flags set on them. E.g. # ls -la /proc/PID/fd lrwx-- 1 root qemu 64 Apr 24 20:14 12 -> /path/to/harddisk.hdd # virsh q

Re: [BUG, RFC] Base node is in RW after making external snapshot

2024-05-10 Thread Andrey Drobyshev
On 4/24/24 21:00, Andrey Drobyshev wrote: > Hi everyone, > > When making an external snapshot, we end up in a situation when 2 block > graph nodes related to the same image file (format and storage nodes) > have different RO flags set on them. > > E.g. > > # ls -la

[PATCH v2 08/11] qcow2: zeroize the entire cluster when there're no non-zero subclusters

2024-05-12 Thread Andrey Drobyshev
When zeroizing the last non-zero subclusters within single cluster, it makes sense to go zeroize the entire cluster and go down zero_in_l2_slice() path right away. That way we'd also update the corresponding refcount table. Signed-off-by: Andrey Drobyshev Reviewed-by: Hanna Czenczek ---

[PATCH v2 06/11] iotests/290: add test case to check 'discard-no-unref' option behavior

2024-05-12 Thread Andrey Drobyshev
) and that with the option enabled cluster is still marked as allocated in "qemu-img map" output. We also check that the option doesn't work with qcow2 v2 images. Signed-off-by: Andrey Drobyshev --- tests/qemu-iotests/290 | 34 ++ tests/

[PATCH v2 02/11] qcow2: simplify L2 entries accounting for discard-no-unref

2024-05-12 Thread Andrey Drobyshev
atter whether we unmap or zeroize the cluster. For that OR'ing with the old entry is enough. This patch doesn't change the logic and is pure refactoring. Signed-off-by: Andrey Drobyshev --- block/qcow2-cluster.c | 34 +++--- 1 file changed, 15 insertions(+

[PATCH v2 09/11] qcow2: make subclusters discardable

2024-05-12 Thread Andrey Drobyshev
with subclusters enabled. Also rename qcow2_cluster_discard() -> qcow2_subcluster_discard() to reflect the change. Signed-off-by: Andrey Drobyshev --- block/qcow2-cluster.c | 106 + block/qcow2-snapshot.c | 6 +-- block/qcow2.c | 25 +

[PATCH v2 01/11] qcow2: make function update_refcount_discard() global

2024-05-12 Thread Andrey Drobyshev
We are going to need it for discarding separate subclusters. The function itself doesn't do anything with the refcount tables, it simply adds a discard request to the queue, so rename it to qcow2_queue_discard(). Signed-off-by: Andrey Drobyshev Reviewed-by: Hanna Czenczek --- block/

[PATCH v2 11/11] iotests/271: add test cases for subcluster-based discard/unmap

2024-05-12 Thread Andrey Drobyshev
make all discard/unmap operations enable trace point 'file_do_fallocate' so that actual fallocate() calls are visible. Signed-off-by: Andrey Drobyshev --- tests/qemu-iotests/271 | 70 +- tests/qemu-iotests/271.out | 69 ++

[PATCH v2 07/11] qcow2: add get_sc_range_info() helper for working with subcluster ranges

2024-05-12 Thread Andrey Drobyshev
ation. Also introduce struct SubClusterRangeInfo, which would contain all the needed params. Signed-off-by: Andrey Drobyshev --- block/qcow2-cluster.c | 140 -- 1 file changed, 108 insertions(+), 32 deletions(-) diff --git a/block/qcow2-cluster.c b/block/

[PATCH v2 05/11] iotests/common.rc: add disk_usage function

2024-05-12 Thread Andrey Drobyshev
Move the definition from iotests/250 to common.rc. This is used to detect real disk usage of sparse files. In particular, we want to use it for checking subclusters-based discards. Signed-off-by: Andrey Drobyshev --- tests/qemu-iotests/250 | 5 - tests/qemu-iotests/common.rc | 6

[PATCH v2 10/11] qcow2: zero_l2_subclusters: fall through to discard operation when requested

2024-05-12 Thread Andrey Drobyshev
quot; will lead to actual unmap. Signed-off-by: Andrey Drobyshev --- block/qcow2-cluster.c | 26 -- 1 file changed, 20 insertions(+), 6 deletions(-) diff --git a/block/qcow2-cluster.c b/block/qcow2-cluster.c index 3c134a7e80..53e04eff93 100644 --- a/block/qcow2-cl

[PATCH v2 03/11] qcow2: put discard requests in the common queue when discard-no-unref enabled

2024-05-12 Thread Andrey Drobyshev
check for the presence of external data files for us and redirect request to underlying data_file. Here we want to do the same but avoid refcount updates, thus we perform the same checks. Suggested-by: Hanna Czenczek Signed-off-by: Andrey Drobyshev --- block/qcow2-cluster.c

[PATCH v2 04/11] block/file-posix: add trace event for fallocate() calls

2024-05-12 Thread Andrey Drobyshev
This would ease debugging of write zeroes and discard operations. Signed-off-by: Andrey Drobyshev --- block/file-posix.c | 1 + block/trace-events | 1 + 2 files changed, 2 insertions(+) diff --git a/block/file-posix.c b/block/file-posix.c index 35684f7e21..45134f0eef 100644 --- a/block/file

[PATCH v2 00/11] qcow2: make subclusters discardable

2024-05-12 Thread Andrey Drobyshev
v1: https://lists.nongnu.org/archive/html/qemu-devel/2023-10/msg07223.html Andrey Drobyshev (11): qcow2: make function update_refcount_discard() global qcow2: simplify L2 entries accounting for discard-no-unref qcow2: put discard requests in the common queue when discard-no-unref

Re: [PATCH 0/6] qemu-img: rebase: add compression support

2023-07-24 Thread Andrey Drobyshev
On 6/30/23 13:54, Denis V. Lunev wrote: > On 6/1/23 21:28, Andrey Drobyshev wrote: >> This series is adding [-c | --compress] option to "qemu-img rebase" >> command, which might prove useful for saving some disk space when, for >> instance, manipulating chains of

Re: [PATCH v2 0/3] block: align CoR requests to subclusters

2023-07-24 Thread Andrey Drobyshev
On 7/11/23 20:25, Andrey Drobyshev wrote: > v1 --> v2: > * Fixed line indentation; > * Fixed wording in a comment; > * Added R-b. > > v1: https://lists.nongnu.org/archive/html/qemu-block/2023-06/msg00606.html > > Andrey Drobyshev (3): > block: add subcluster

Re: [PATCH v2 0/3] qemu-img: map: implement support for compressed clusters

2023-07-24 Thread Andrey Drobyshev
On 7/6/23 19:30, Andrey Drobyshev wrote: > v1 --> v2: > * Add vmdk format to the 1st commit. Tweak commit message accordingly; > * Make "compressed" field in MapEntry optional. > > v1: https://lists.nongnu.org/archive/html/qemu-block/2023-06/msg00184.html > &g

Re: [PATCH 0/6] qemu-img: rebase: add compression support

2023-07-31 Thread Andrey Drobyshev
On 7/24/23 16:11, Andrey Drobyshev wrote: > On 6/30/23 13:54, Denis V. Lunev wrote: >> On 6/1/23 21:28, Andrey Drobyshev wrote: >>> This series is adding [-c | --compress] option to "qemu-img rebase" >>> command, which might prove useful for saving s

Re: [PATCH v2 0/3] qemu-img: map: implement support for compressed clusters

2023-07-31 Thread Andrey Drobyshev
On 7/24/23 16:10, Andrey Drobyshev wrote: > On 7/6/23 19:30, Andrey Drobyshev wrote: >> v1 --> v2: >> * Add vmdk format to the 1st commit. Tweak commit message accordingly; >> * Make "compressed" field in MapEntry optional. >> >> v1: https://li

Re: [PATCH v2 0/3] block: align CoR requests to subclusters

2023-07-31 Thread Andrey Drobyshev
On 7/24/23 16:11, Andrey Drobyshev wrote: > On 7/11/23 20:25, Andrey Drobyshev wrote: >> v1 --> v2: >> * Fixed line indentation; >> * Fixed wording in a comment; >> * Added R-b. >> >> v1: https://lists.nongnu.org/archive/html/qemu-block/2023-

Re: [PATCH v2 0/3] block: align CoR requests to subclusters

2023-08-16 Thread Andrey Drobyshev
On 7/31/23 17:51, Andrey Drobyshev wrote: > On 7/24/23 16:11, Andrey Drobyshev wrote: >> On 7/11/23 20:25, Andrey Drobyshev wrote: >>> v1 --> v2: >>> * Fixed line indentation; >>> * Fixed wording in a comment; >>> * Added R-b. >>> >>

Re: [PATCH v2 0/3] qemu-img: map: implement support for compressed clusters

2023-08-16 Thread Andrey Drobyshev
On 7/31/23 17:45, Andrey Drobyshev wrote: > On 7/24/23 16:10, Andrey Drobyshev wrote: >> On 7/6/23 19:30, Andrey Drobyshev wrote: >>> v1 --> v2: >>> * Add vmdk format to the 1st commit. Tweak commit message accordingly; >>> * Make "compressed&qu

Re: [PATCH 0/6] qemu-img: rebase: add compression support

2023-08-16 Thread Andrey Drobyshev
On 7/31/23 17:43, Andrey Drobyshev wrote: > On 7/24/23 16:11, Andrey Drobyshev wrote: >> On 6/30/23 13:54, Denis V. Lunev wrote: >>> On 6/1/23 21:28, Andrey Drobyshev wrote: >>>> This series is adding [-c | --compress] option to "qemu-img rebase" >>&

Re: [PATCH v2 0/3] block: align CoR requests to subclusters

2023-08-22 Thread Andrey Drobyshev
On 8/16/23 12:22, Andrey Drobyshev wrote: > On 7/31/23 17:51, Andrey Drobyshev wrote: >> On 7/24/23 16:11, Andrey Drobyshev wrote: >>> On 7/11/23 20:25, Andrey Drobyshev wrote: >>>> v1 --> v2: >>>> * Fixed line indentation; >>>> * Fi

Re: [PATCH 0/6] qemu-img: rebase: add compression support

2023-08-22 Thread Andrey Drobyshev
On 8/16/23 12:22, Andrey Drobyshev wrote: > On 7/31/23 17:43, Andrey Drobyshev wrote: >> On 7/24/23 16:11, Andrey Drobyshev wrote: >>> On 6/30/23 13:54, Denis V. Lunev wrote: >>>> On 6/1/23 21:28, Andrey Drobyshev wrote: >>>>> This series is adding

Re: [PATCH v2 0/3] qemu-img: map: implement support for compressed clusters

2023-08-22 Thread Andrey Drobyshev
On 8/16/23 12:22, Andrey Drobyshev wrote: > On 7/31/23 17:45, Andrey Drobyshev wrote: >> On 7/24/23 16:10, Andrey Drobyshev wrote: >>> On 7/6/23 19:30, Andrey Drobyshev wrote: >>>> v1 --> v2: >>>> * Add vmdk format to the 1st commit. Tweak commit m

Re: [PATCH v2 0/3] block: align CoR requests to subclusters

2023-08-23 Thread Andrey Drobyshev
On 8/22/23 22:58, John Snow wrote: > On Tue, Aug 22, 2023 at 1:33 PM Andrey Drobyshev > wrote: >> >> On 8/16/23 12:22, Andrey Drobyshev wrote: >>> On 7/31/23 17:51, Andrey Drobyshev wrote: >>>> On 7/24/23 16:11, Andrey Drobyshev wrote: >>>>&g

Re: [PATCH v2 0/3] block: align CoR requests to subclusters

2023-08-24 Thread Andrey Drobyshev
On 8/24/23 17:32, Stefan Hajnoczi wrote: > On Wed, Aug 23, 2023 at 03:50:55PM +0300, Andrey Drobyshev wrote: >> On 8/22/23 22:58, John Snow wrote: >>> On Tue, Aug 22, 2023 at 1:33 PM Andrey Drobyshev >>> wrote: >>>> >>>> On 8/16/23 12:22, And

Re: [PATCH v2 2/3] qemu-img: map: report compressed data blocks

2023-08-28 Thread Andrey Drobyshev
On 8/25/23 17:14, Hanna Czenczek wrote: > On 06.07.23 18:30, Andrey Drobyshev wrote: >> Right now "qemu-img map" reports compressed blocks as containing data >> but having no host offset.  This is not very informative.  Instead, >> let's add another boolean f

Re: [PATCH 3/6] qemu-img: rebase: use backing files' BlockBackend for buffer alignment

2023-08-29 Thread Andrey Drobyshev
On 8/25/23 17:29, Hanna Czenczek wrote: > On 01.06.23 21:28, Andrey Drobyshev via wrote: >> Since commit bb1c05973cf ("qemu-img: Use qemu_blockalign"), buffers for >> the data read from the old and new backing files are aligned using >> BlockDriverState (or BlockBac

Re: [PATCH 4/6] qemu-img: rebase: avoid unnecessary COW operations

2023-08-29 Thread Andrey Drobyshev
On 8/25/23 18:00, Hanna Czenczek wrote: > On 01.06.23 21:28, Andrey Drobyshev via wrote: >> When rebasing an image from one backing file to another, we need to >> compare data from old and new backings.  If the diff between that data >> happens to be unaligned to the target c

Re: [PATCH v2 3/3] tests/qemu-iotests/197: add testcase for CoR with subclusters

2023-09-07 Thread Andrey Drobyshev
On 9/6/23 12:43, Denis V. Lunev wrote: > On 7/11/23 19:25, Andrey Drobyshev wrote: >> Add testcase which checks that allocations during copy-on-read are >> performed on the subcluster basis when subclusters are enabled in target >> image. >> >> This testcase also t

Re: [PATCH v3 2/2] qemu-img: map: report compressed data blocks

2023-09-14 Thread Andrey Drobyshev
On 9/15/23 00:17, Eric Blake wrote: > On Fri, Sep 08, 2023 at 12:02:26AM +0300, Andrey Drobyshev wrote: >> Right now "qemu-img map" reports compressed blocks as containing data >> but having no host offset. This is not very informative. Instead, >> let&#x

Re: [PATCH v2 3/8] qemu-img: rebase: use backing files' BlockBackend for buffer alignment

2023-09-15 Thread Andrey Drobyshev
On 9/15/23 21:39, Eric Blake wrote: > On Fri, Sep 15, 2023 at 07:20:11PM +0300, Andrey Drobyshev wrote: >> Since commit bb1c05973cf ("qemu-img: Use qemu_blockalign"), buffers for >> the data read from the old and new backing files are aligned using >> BlockDriver

Re: [PATCH v2 5/8] qemu-img: rebase: avoid unnecessary COW operations

2023-09-16 Thread Andrey Drobyshev
On 9/16/23 00:52, Eric Blake wrote: > On Fri, Sep 15, 2023 at 07:20:13PM +0300, Andrey Drobyshev wrote: >> When rebasing an image from one backing file to another, we need to >> compare data from old and new backings. If the diff between that data >> happens to be unaligned

Re: [PATCH v2 3/8] qemu-img: rebase: use backing files' BlockBackend for buffer alignment

2023-09-19 Thread Andrey Drobyshev
On 9/19/23 11:18, Hanna Czenczek wrote: > On 15.09.23 18:20, Andrey Drobyshev wrote: >> Since commit bb1c05973cf ("qemu-img: Use qemu_blockalign"), buffers for >> the data read from the old and new backing files are aligned using >> BlockDriverState (or BlockBac

  1   2   >