[PATCH v4 2/3] nbd: Set unix socket send buffer on macOS

2025-05-17 Thread Nir Soffer
ket=/tmp/nbd.sock" | buffer size | time| user| system | |-|-|-|-| | default | 13.085 | 5.664 | 6.461 | | 65536 | 3.299 | 5.106 | 2.515 | | 131072 | 2.396 | 4.989 | 2.069 | | 262144 | 1.607 | 4.724 | 1.555 | |

[PATCH v4 1/3] io: Add helper for setting socket send buffer size

2025-05-17 Thread Nir Soffer
don't need a helper for receive buffer size since it is not used with unix domain sockets. This is documented for Linux, and not documented for macOS. Failing to set the socket buffer size is not a fatal error, but the caller may want to warn about the failure. Signed-off-by: Nir S

[PATCH v4 0/3] nbd: Increase unix socket buffer size

2025-05-17 Thread Nir Soffer
check for SOCK_STREAM since NBD does not support SOCK_DGRAM. - Warn if we fail to set the socket send buffer size. v3 was here: https://lists.gnu.org/archive/html/qemu-devel/2025-04/msg05096.html Nir Soffer (3): io: Add helper for setting socket send buffer size nbd: Set unix socket send buffer

[PATCH v4 3/3] nbd: Set unix socket send buffer on Linux

2025-05-17 Thread Nir Soffer
| |-|-|-|-| | default | 3.582 | 4.595 | 2.392 | | 524288 | 1.499 | 4.384 | 1.482 | | 1048576 | 1.377 | 4.381 | 1.345 | | 2097152 | 1.388 | 4.389 | 1.354 | | 4194304 | 1.352 | 4.395 | 1.302 | Signed-off-by: Nir Soffer --- nbd/common.c | 3 ++- 1 file changed, 2

Re: [PATCH v2 1/2] block/null: Report DATA if not reading zeroes

2025-05-16 Thread Nir Soffer
> On 8 May 2025, at 7:37, Markus Armbruster wrote: > > Nir Soffer writes: > >> If read-zeroes is not set, we did not report BDRV_BLOCK_DATA or >> BDRV_BLOCK_ZERO. This is not consistent with other drivers and can >> confuse users or other programs: >>

Re: [PATCH v2 1/2] block/null: Report DATA if not reading zeroes

2025-05-16 Thread Nir Soffer
> On 8 May 2025, at 8:20, Markus Armbruster wrote: > > Markus Armbruster writes: > >> Markus Armbruster writes: >> >>> Nir Soffer writes: > > [...] > >>>> diff --git a/qapi/block-core.json b/qapi/block-core.json >>>> i

Re: [PATCH v2 1/2] block/null: Report DATA if not reading zeroes

2025-05-16 Thread Nir Soffer
> On 8 May 2025, at 8:03, Markus Armbruster wrote: > > Markus Armbruster writes: > >> Nir Soffer writes: >> >>> If read-zeroes is not set, we did not report BDRV_BLOCK_DATA or >>> BDRV_BLOCK_ZERO. This is not consistent with other drivers a

Re: [PATCH v2 2/2] block/null: Add read-pattern option

2025-05-16 Thread Nir Soffer
> On 8 May 2025, at 8:28, Markus Armbruster wrote: > > Nir Soffer writes: > >> When the `read-zeroes` is set, reads produce zeroes, and block status >> return BDRV_BLOCK_ZERO, emulating a sparse image. >> >> If we don't set `read-zeros` we rep

Re: [PATCH v3 1/2] io: Increase unix socket buffers size on macOS

2025-05-07 Thread Nir Soffer
> On 7 May 2025, at 19:37, Daniel P. Berrangé wrote: > > On Sun, Apr 27, 2025 at 07:50:28PM +0300, Nir Soffer wrote: >> On macOS we need to increase unix stream socket buffers size on the >> client and server to get good performance. We set socket buffers on >>

Re: [PATCH v3 2/2] io: Increase unix socket buffers on Linux

2025-04-30 Thread Nir Soffer
> On 29 Apr 2025, at 0:37, Eric Blake wrote: > > On Sun, Apr 27, 2025 at 07:50:29PM +0300, Nir Soffer wrote: >> Like macOS we have similar issue on Linux. For TCP socket the send >> buffer size is 2626560 bytes (~2.5 MiB) and we get good performance. >> However for

[PATCH v2 1/2] block/null: Report DATA if not reading zeroes

2025-04-30 Thread Nir Soffer
river': 'raw', 'file': {'driver': 'null-co', 'size': '1g'}}" [{ "start": 0, "length": 1073741824, "depth": 0, "present": true, "zero": false, "data": true, "compres

[PATCH v2 0/2] block/null: Add read-pattern option

2025-04-30 Thread Nir Soffer
/2025-04/msg05098.html Nir Soffer (2): block/null: Report DATA if not reading zeroes block/null: Add read-pattern option block/null.c | 38 +++--- docs/devel/secure-coding-practices.rst | 3 +- qapi/block-core.json | 17

[PATCH v2 2/2] block/null: Add read-pattern option

2025-04-30 Thread Nir Soffer
, 'file': {'driver': 'null-co', 'read-pattern': -1}}" qemu-img: Could not open 'json:{...}': read_pattern is out of range (0-255) % ./qemu-img map --output json \ "json:{'driver': 'raw', 'fil

Re: [PATCH 2/2] block/null: Add read-pattern option

2025-04-30 Thread Nir Soffer
> On 29 Apr 2025, at 0:55, Eric Blake wrote: > > On Mon, Apr 28, 2025 at 01:59:00AM +0300, Nir Soffer wrote: >> When the `read-zeroes` is set, reads produce zeroes, and block status >> return BDRV_BLOCK_ZERO, emulating a sparse image. >> >> If we

Re: [PATCH 2/2] block/null: Add read-pattern option

2025-04-30 Thread Nir Soffer
> On 29 Apr 2025, at 9:00, Markus Armbruster wrote: > > Eric Blake writes: > >> On Mon, Apr 28, 2025 at 01:59:00AM +0300, Nir Soffer wrote: >>> When the `read-zeroes` is set, reads produce zeroes, and block status >>> return BDRV_BLOCK_ZERO, emulating

Re: [PATCH 1/2] block/null: Report DATA if not reading zeroes

2025-04-30 Thread Nir Soffer
> On 29 Apr 2025, at 0:49, Eric Blake wrote: > > On Mon, Apr 28, 2025 at 01:58:59AM +0300, Nir Soffer wrote: >> If read-zeroes is not set, we did not report BDRV_BLOCK_DATA or >> BDRV_BLOCK_ZERO. This is not consistent with other drivers and can >> conf

[PATCH 1/2] block/null: Report DATA if not reading zeroes

2025-04-27 Thread Nir Soffer
river': 'raw', 'file': {'driver': 'null-co', 'size': '1g'}}" [{ "start": 0, "length": 1073741824, "depth": 0, "present": true, "zero": false, "data": true, "compres

[PATCH 2/2] block/null: Add read-pattern option

2025-04-27 Thread Nir Soffer
ern': -1}}" & % time blksum 'nbd+unix:///?socket=/tmp/data.sock' 2c122b3ed28c83ede3c08485659fa9b56ee54ba1751db74d8ba9aa13d9866432 nbd+unix:///?socket=/tmp/data.sock blksum 'nbd+unix:///?socket=/tmp/data.sock' 46.05s user 14.15s system 448% cpu 13.414

[PATCH 0/2] block/null: Add read-pattern

2025-04-27 Thread Nir Soffer
image full of zeroes and image for of non-zero bytes. I used this for testing https://lists.gnu.org/archive/html/qemu-devel/2025-04/msg05096.html Nir Soffer (2): block/null: Report DATA if not reading zeroes block/null: Add read-pattern option block/null.c | 21

[PATCH v3 0/2] io: Increase unix stream socket buffer size

2025-04-27 Thread Nir Soffer
- datagram socket need different configuration - Modify the code to make it easy to support unix datagram socket v2 was here: https://lists.gnu.org/archive/html/qemu-devel/2025-04/msg03167.html Nir Soffer (2): io: Increase unix socket buffers size on macOS io: Increase unix socket buffers on Linux

[PATCH v3 2/2] io: Increase unix socket buffers on Linux

2025-04-27 Thread Nir Soffer
| |-|-|-|-| | default | 3.582 | 4.595 | 2.392 | | 524288 | 1.499 | 4.384 | 1.482 | | 1048576 | 1.377 | 4.381 | 1.345 | | 2097152 | 1.388 | 4.389 | 1.354 | | 4194304 | 1.352 | 4.395 | 1.302 | Signed-off-by: Nir Soffer --- io/channel-socket.c | 9 + 1 file

[PATCH v3 1/2] io: Increase unix socket buffers size on macOS

2025-04-27 Thread Nir Soffer
72 | 2.396 | 4.989 | 2.069 | | 262144 | 1.607 | 4.724 | 1.555 | | 524288 | 1.271 | 4.528 | 1.224 | | 1048576 | 1.294 | 4.565 | 1.333 | | 2097152 | 1.299 | 4.569 | 1.344 | | 4194304 | 1.291 | 4.559 | 1.327 | Signed-off-by: Nir Soffer --- io/

Re: [PATCH v2] io: Increase unix socket buffers size on macOS

2025-04-22 Thread Nir Soffer
> On 22 Apr 2025, at 13:42, Daniel P. Berrangé wrote: > > On Sun, Apr 20, 2025 at 02:12:18AM +0300, Nir Soffer wrote: >> On macOS we need to increase unix socket buffers size on the client and >> server to get good performance. We set the socket buffers on macOS a

[PATCH v2] io: Increase unix socket buffers size on macOS

2025-04-19 Thread Nir Soffer
world tests larger buffers are optimal, so I picked send buffer of 1m and receive buffer of 4m. This will improve other usage of unix domain sockets on macOS. I tested only reading from qemu-nbd. The same change for libnbd: https://gitlab.com/nbdkit/libnbd/-/merge_requests/21 Signed-off-by: Nir

Re: [PATCH] io: Set unix socket buffers on macOS

2025-04-18 Thread Nir Soffer
> On 18 Apr 2025, at 21:55, Eric Blake wrote: > > On Fri, Apr 18, 2025 at 05:24:36PM +0300, Nir Soffer wrote: >> Testing with qemu-nbd shows that computing a hash of an image via >> qemu-nbd is 5-7 times faster with this change. >> > >> +++ b/io/c

Re: [PATCH] io: Set unix socket buffers on macOS

2025-04-18 Thread Nir Soffer
This should be changed also on the client side. The libnbd part is here: https://gitlab.com/nbdkit/libnbd/-/merge_requests/21 We may want to change also the nbd client code used in qemu-img. I can look at this later. > On 18 Apr 2025, at 17:24, Nir Soffer wrote: > > Testing with

Re: [PATCH] io: Set unix socket buffers on macOS

2025-04-18 Thread Nir Soffer
> On 18 Apr 2025, at 17:50, Philippe Mathieu-Daudé wrote: > > Hi Nir, > > On 18/4/25 16:24, Nir Soffer wrote: >> Testing with qemu-nbd shows that computing a hash of an image via >> qemu-nbd is 5-7 times faster with this change. >> Tested with 2 qemu-nbd pro

[PATCH] io: Set unix socket buffers on macOS

2025-04-18 Thread Nir Soffer
Time (mean ± σ): 1.946 s ± 0.017 s[User: 4.541 s, System: 1.481 s] Range (min … max):1.912 s … 1.979 s10 runs Summary blksum 'nbd+unix:///?socket=/tmp/after.sock' ran 6.99 ± 0.07 times faster than blksum 'nbd+unix:///?socket=/tmp/before.sock&

[PATCH v2] iotest: Unbreak 302 with python 3.13

2025-02-28 Thread Nir Soffer
.offset points to the start of the member data instead of the end. [1] https://github.com/python/cpython/pull/117988 Signed-off-by: Nir Soffer --- tests/qemu-iotests/302 | 19 ++- 1 file changed, 14 insertions(+), 5 deletions(-) diff --git a/tests/qemu-iotests/302 b/tests/

Re: qemu-iotests 302 failing

2025-02-03 Thread Nir Soffer
> On 3 Feb 2025, at 20:26, Stefan Hajnoczi wrote: > > Hi Nir, > qemu-iotests 302 is failing: > > $ git show > commit 6fccaa2fba391815308a746d68f7fa197bc93586 > ... > $ python --version # on Fedora 41 x86_64 > Python 3.13.1 > $ tests/qemu-iotests/check -qcow2 302 > 302 fail [13:1

Re: [PATCH v2] scripts/qcow2-to-stdout.py: Add script to write qcow2 images to stdout

2024-07-27 Thread Nir Soffer
On Mon, Jul 1, 2024 at 6:13 PM Alberto Garcia wrote: > > This tool converts a disk image to qcow2, writing the result directly > to stdout. This can be used for example to send the generated file > over the network. > > This is equivalent to using qemu-img to convert a file to qcow2 and > then wri

Re: [PATCH 3/4] iotests: Change imports for Python 3.13

2024-07-02 Thread Nir Soffer
> On 2 Jul 2024, at 17:44, John Snow wrote: > > > > On Tue, Jul 2, 2024 at 7:52 AM Nir Soffer <mailto:nsof...@redhat.com>> wrote: >> On Thu, Jun 27, 2024 at 2:23 AM John Snow > <mailto:js...@redhat.com>> wrote: >> > >> > Python 3.13

Re: [PATCH 3/4] iotests: Change imports for Python 3.13

2024-07-02 Thread Nir Soffer
On Thu, Jun 27, 2024 at 2:23 AM John Snow wrote: > > Python 3.13 isn't out yet, but it's in beta and Fedora is ramping up to > make it the default system interpreter for Fedora 41. > > They moved our cheese for where ContextManager lives; add a conditional > to locate it while we support both pre-

[PATCH v3 0/2] Consider discard option when writing zeros

2024-06-28 Thread Nir Soffer
was here: https://lists.nongnu.org/archive/html/qemu-block/2024-06/msg00231.html Nir Soffer (2): qemu-iotest/245: Add missing discard=unmap Consider discard option when writing zeros block/io.c| 9 +- tests/qemu-iotests/245| 2

[PATCH v3 1/2] qemu-iotest/245: Add missing discard=unmap

2024-06-28 Thread Nir Soffer
The test works since we punch holes by default even when opening the image without discard=on or discard=unmap. Fix the test to enable discard. --- tests/qemu-iotests/245 | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/tests/qemu-iotests/245 b/tests/qemu-iotests/245 index a934c

[PATCH v3 2/2] Consider discard option when writing zeros

2024-06-28 Thread Nir Soffer
u.org/archive/html/qemu-discuss/2024-06/msg3.html Signed-off-by: Nir Soffer --- block/io.c| 9 +- tests/qemu-iotests/tests/write-zeroes-unmap | 127 ++ .../qemu-iotests/tests/write-zeroes-unmap.out | 81 +++ 3 files changed

Re: [PATCH v2] Consider discard option when writing zeros

2024-06-28 Thread Nir Soffer
On Thu, Jun 27, 2024 at 2:42 PM Kevin Wolf wrote: > Am 26.06.2024 um 18:27 hat Nir Soffer geschrieben: > > On Wed, Jun 26, 2024 at 12:17 PM Daniel P. Berrangé > > > wrote: > > > > > On Mon, Jun 24, 2024 at 06:08:26PM +0200, Kevin Wolf wrote: > > > &g

Re: [PATCH v2] Consider discard option when writing zeros

2024-06-26 Thread Nir Soffer
On Wed, Jun 26, 2024 at 12:17 PM Daniel P. Berrangé wrote: > On Mon, Jun 24, 2024 at 06:08:26PM +0200, Kevin Wolf wrote: > > Am 24.06.2024 um 17:23 hat Stefan Hajnoczi geschrieben: > > > On Wed, Jun 19, 2024 at 08:43:25PM +0300, Nir Soffer wrote: > > > > Te

Re: [PATCH v2] Consider discard option when writing zeros

2024-06-26 Thread Nir Soffer
On Wed, Jun 26, 2024 at 11:42 AM Kevin Wolf wrote: > Am 24.06.2024 um 23:12 hat Nir Soffer geschrieben: > > On Mon, Jun 24, 2024 at 7:08 PM Kevin Wolf wrote: > > > > > Am 24.06.2024 um 17:23 hat Stefan Hajnoczi geschrieben: > > > > On Wed, Jun 19, 2024 a

Re: [PATCH v2] Consider discard option when writing zeros

2024-06-24 Thread Nir Soffer
On Mon, Jun 24, 2024 at 7:08 PM Kevin Wolf wrote: > Am 24.06.2024 um 17:23 hat Stefan Hajnoczi geschrieben: > > On Wed, Jun 19, 2024 at 08:43:25PM +0300, Nir Soffer wrote: > > > Tested using: > > > > Hi Nir, > > This looks like a good candidate for the qemu-

Re: [PATCH v2] Consider discard option when writing zeros

2024-06-19 Thread Nir Soffer
On Wed, Jun 19, 2024 at 8:40 PM Nir Soffer wrote: > - Need to run all block tests > Stale note, make check pass

Re: [PATCH v2] Consider discard option when writing zeros

2024-06-19 Thread Nir Soffer
est.raw discard=unmap - write zeroes unmap 0 /tmp/test.raw Differences: $ diff -u before.out after.out --- before.out 2024-06-19 20:24:09.234083713 +0300 +++ after.out 2024-06-19 20:24:20.526165573 +0300 @@ -3,13 +3,13 @@ 1.0M /tmp/test.raw defaults - write zeroes unmap -0 /tmp/test.raw +

[PATCH v2] Consider discard option when writing zeros

2024-06-19 Thread Nir Soffer
his change we punched holes by default. I'm not sure this is a good change in behavior. - Need to run all block tests - Not sure that we have tests covering unmapping, we may need new tests - We may need new tests to cover this change Signed-off-by: Nir Soffer --- Changes since v1: - Replac

Re: [PATCH] block/file-posix: Consider discard flag when opening

2024-06-19 Thread Nir Soffer
> On 19 Jun 2024, at 11:16, Kevin Wolf wrote: > > Am 18.06.2024 um 23:24 hat Nir Soffer geschrieben: >> Set has_discard only when BDRV_O_UNMAP is not set. With this users that >> want to keep their images fully allocated can disable hole punching >> when writin

[PATCH] block/file-posix: Consider discard flag when opening

2024-06-18 Thread Nir Soffer
un by default. We needs tests for keeping images non-sparse. Signed-off-by: Nir Soffer --- block/file-posix.c | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/block/file-posix.c b/block/file-posix.c index be25e35ff6..acac2abadc 100644 --- a/block/file-posix.c +++ b/block

Re: [PATCH 1/1] block: improve alignment detection and fix 271 test

2023-10-14 Thread Nir Soffer
e test to reflect real world usage? _reset_img 2083k I guess it works with: _reset_img 2084k Commits > commit a6b257a08e3d72219f03e461a52152672fec0612 > Author: Nir Soffer > Date: Tue Aug 13 21:21:03 2019 +0300 > file-posix: Handle undetectable alignme

[PATCH] libvhost-user: Fix update of signalled_used

2023-05-09 Thread Nir Soffer
d-off-by: Nir Soffer --- subprojects/libvhost-user/libvhost-user.c | 23 +-- 1 file changed, 17 insertions(+), 6 deletions(-) diff --git a/subprojects/libvhost-user/libvhost-user.c b/subprojects/libvhost-user/libvhost-user.c index 8fb61e2df2..5f26d2d378 100644 --- a/sub

Re: [Libguestfs] [PATCH v2 1/6] spec: Recommend cap on NBD_REPLY_TYPE_BLOCK_STATUS length

2023-03-06 Thread Nir Soffer
On Sun, Mar 5, 2023 at 10:42 AM Wouter Verhelst wrote: > > On Fri, Mar 03, 2023 at 04:17:40PM -0600, Eric Blake wrote: > > On Fri, Dec 16, 2022 at 10:32:01PM +0300, Vladimir Sementsov-Ogievskiy > > wrote: > > > s-o-b line missed. > > > > I'm not sure if the NBD project has a strict policy on incl

Re: [PATCH v2 2/5] Support format or cache specific out file

2022-12-13 Thread Nir Soffer
On Tue, Dec 13, 2022 at 8:09 PM Hanna Reitz wrote: > > On 13.12.22 16:56, Nir Soffer wrote: > > On Mon, Dec 12, 2022 at 12:38 PM Hanna Reitz wrote: > >> On 28.11.22 15:15, Nir Soffer wrote: > >>> Extend the test finder to find tests with format (*.o

Re: [PATCH v2 2/5] Support format or cache specific out file

2022-12-13 Thread Nir Soffer
On Mon, Dec 12, 2022 at 12:38 PM Hanna Reitz wrote: > > On 28.11.22 15:15, Nir Soffer wrote: > > Extend the test finder to find tests with format (*.out.qcow2) or cache > > specific (*.out.nocache) out file. This worked before only for the > > numbered tests. > >

[PATCH v2 3/5] qemu-img: Add checksum command

2022-11-28 Thread Nir Soffer
blkhash/ [3] Computing checksum for 8T empty image: qemu-img checksum: 3.7s, sha256sum (estimate): 17,749s Signed-off-by: Nir Soffer --- docs/tools/qemu-img.rst | 24 ++ meson.build | 10 ++- meson_options.txt | 2 + qemu-img-cmds.hx| 8 ++ qemu-img.c

[PATCH v2 0/5] Add qemu-img checksum command using blkhash

2022-11-28 Thread Nir Soffer
tml - https://lists.nongnu.org/archive/html/qemu-block/2022-11/msg00171.html - https://lists.nongnu.org/archive/html/qemu-block/2022-11/msg00173.html Nir Soffer (5): qemu-img.c: Move IO_BUF_SIZE to the top of the file Support format or cache specific out file qemu-img: Add checksum command iote

[PATCH v2 5/5] qemu-img: Speed up checksum

2022-11-28 Thread Nir Soffer
5s ±0.183s | 1.808s ±0.016s | x1.36 | | nbd | 6g | direct| 3.540s ±0.020s | 1.749s ±0.018s | x2.02 | [1] raw image full of zeroes [2] raw fedora 35 image with additional random data, 50% full [3] image [2] exported by qemu-nbd via unix socket Signed-off-by: Nir Soffer --- qemu-i

[PATCH v2 4/5] iotests: Test qemu-img checksum

2022-11-28 Thread Nir Soffer
help debugging in case of failures, the output includes a json map of the test image. Signed-off-by: Nir Soffer --- tests/qemu-iotests/tests/qemu-img-checksum| 63 +++ .../tests/qemu-img-checksum.out.qcow2 | 11 .../tests/qemu-img-checksum.out.raw | 10 +

[PATCH v2 2/5] Support format or cache specific out file

2022-11-28 Thread Nir Soffer
Extend the test finder to find tests with format (*.out.qcow2) or cache specific (*.out.nocache) out file. This worked before only for the numbered tests. --- tests/qemu-iotests/findtests.py | 10 -- 1 file changed, 8 insertions(+), 2 deletions(-) diff --git a/tests/qemu-iotests/findtests

[PATCH v2 1/5] qemu-img.c: Move IO_BUF_SIZE to the top of the file

2022-11-28 Thread Nir Soffer
This macro is used by various commands (compare, convert, rebase) but it is defined somewhere in the middle of the file. I'm going to use it in the new checksum command so lets clean up a bit before that. --- qemu-img.c | 3 +-- 1 file changed, 1 insertion(+), 2 deletions(-) diff --git a/qemu-img

Re: [PATCH 2/3] iotests: Test qemu-img checksum

2022-11-28 Thread Nir Soffer
On Mon, Nov 7, 2022 at 1:41 PM Hanna Reitz wrote: > On 30.10.22 18:38, Nir Soffer wrote: > > On Wed, Oct 26, 2022 at 4:31 PM Hanna Reitz wrote: > > > > On 01.09.22 16:32, Nir Soffer wrote: > > > Add simple tests creating an image with all kinds

Re: [PATCH 1/3] qemu-img: Add checksum command

2022-11-28 Thread Nir Soffer
On Mon, Nov 7, 2022 at 12:20 PM Hanna Reitz wrote: > On 30.10.22 18:37, Nir Soffer wrote: > > On Wed, Oct 26, 2022 at 4:00 PM Hanna Reitz wrote: > > > > On 01.09.22 16:32, Nir Soffer wrote: > [...] > > > --- > > > docs/tools/qemu-i

Re: [PATCH 3/3] qemu-img: Speed up checksum

2022-10-30 Thread Nir Soffer
On Sun, Oct 30, 2022 at 7:38 PM Nir Soffer wrote: > On Wed, Oct 26, 2022 at 4:54 PM Hanna Reitz wrote: > >> On 01.09.22 16:32, Nir Soffer wrote: >> > [...] > > +/* The current chunk. */ >> > +int64_t offset; >> > +int64_t length; >>

Re: [PATCH 1/3] qemu-img: Add checksum command

2022-10-30 Thread Nir Soffer
On Wed, Oct 26, 2022 at 4:00 PM Hanna Reitz wrote: > On 01.09.22 16:32, Nir Soffer wrote: > > The checksum command compute a checksum for disk image content using the > > blkhash library[1]. The blkhash library is not packaged yet, but it is > > available via copr[2]

Re: [PATCH 3/3] qemu-img: Speed up checksum

2022-10-30 Thread Nir Soffer
On Wed, Oct 26, 2022 at 4:54 PM Hanna Reitz wrote: > On 01.09.22 16:32, Nir Soffer wrote: > > Add coroutine based loop inspired by `qemu-img convert` design. > > > > Changes compared to `qemu-img convert`: > > > > - State for the entire image is kept in Img

Re: [PATCH 2/3] iotests: Test qemu-img checksum

2022-10-30 Thread Nir Soffer
On Wed, Oct 26, 2022 at 4:31 PM Hanna Reitz wrote: > On 01.09.22 16:32, Nir Soffer wrote: > > Add simple tests creating an image with all kinds of extents, different > > formats, different backing chain, different protocol, and different > > image options. Since all image

Re: [PATCH 0/3] Add qemu-img checksum command using blkhash

2022-10-18 Thread Nir Soffer
On Sun, Sep 18, 2022 at 12:35 PM Nir Soffer wrote: > ping > > Kevin, Hanna, I hope you have time to take a look. > > https://lists.nongnu.org/archive/html/qemu-block/2022-09/msg00021.html Ping again, hopefully someone has time to look at this :-) > > > > On Thu,

Re: [PATCH 0/3] Add qemu-img checksum command using blkhash

2022-09-18 Thread Nir Soffer
ping Kevin, Hanna, I hope you have time to take a look. https://lists.nongnu.org/archive/html/qemu-block/2022-09/msg00021.html On Thu, Sep 1, 2022 at 5:32 PM Nir Soffer wrote: > > Since blkhash is available only via copr now, the new command is added as > optional feature, buil

[PATCH 1/3] qemu-img: Add checksum command

2022-09-01 Thread Nir Soffer
blkhash/ [3] Computing checksum for 8T empty image: qemu-img checksum: 3.7s, sha256sum (estimate): 17,749s Signed-off-by: Nir Soffer --- docs/tools/qemu-img.rst | 22 + meson.build | 10 ++- meson_options.txt | 2 + qemu-img-cmds.hx| 8 ++ qemu-img.c

[PATCH 0/3] Add qemu-img checksum command using blkhash

2022-09-01 Thread Nir Soffer
Since blkhash is available only via copr now, the new command is added as optional feature, built only if blkhash-devel package is installed. Nir Soffer (3): qemu-img: Add checksum command iotests: Test qemu-img checksum qemu-img: Speed up checksum docs/tools/qemu-img.rst

[PATCH 3/3] qemu-img: Speed up checksum

2022-09-01 Thread Nir Soffer
5s ±0.183s | 1.808s ±0.016s | x1.36 | | nbd | 6g | direct| 3.540s ±0.020s | 1.749s ±0.018s | x2.02 | [1] raw image full of zeroes [2] raw fedora 35 image with additional random data, 50% full [3] image [2] exported by qemu-nbd via unix socket Signed-off-by: Nir Soffer --- qemu-i

[PATCH 2/3] iotests: Test qemu-img checksum

2022-09-01 Thread Nir Soffer
includes a json map of every test image. Signed-off-by: Nir Soffer --- tests/qemu-iotests/tests/qemu-img-checksum| 149 ++ .../qemu-iotests/tests/qemu-img-checksum.out | 74 + 2 files changed, 223 insertions(+) create mode 100755 tests/qemu-iotests/tests/qemu-img

Re: [PATCH 2/6] virtio-scsi: don't waste CPU polling the event virtqueue

2022-04-27 Thread Nir Soffer
-M accel=kvm -m 1G -cpu host \ > --object iothread,id=iothread0 \ > --device virtio-scsi-pci,iothread=iothread0 \ > --blockdev > file,filename=test.img,aio=native,cache.direct=on,node-name=drive0 \ > --device scsi-hd,drive=drive0 > > After this pat

Re: [PATCH v2] nbd/server: Allow MULTI_CONN for shared writable exports

2022-02-16 Thread Nir Soffer
On Wed, Feb 16, 2022 at 10:08 AM Vladimir Sementsov-Ogievskiy wrote: > > 16.02.2022 02:24, Eric Blake wrote: > > On Tue, Feb 15, 2022 at 09:23:36PM +0200, Nir Soffer wrote: > >> On Tue, Feb 15, 2022 at 7:22 PM Eric Blake wrote: > >> > >>> Accor

Re: [PATCH v2] nbd/server: Allow MULTI_CONN for shared writable exports

2022-02-16 Thread Nir Soffer
On Wed, Feb 16, 2022 at 12:13 PM Richard W.M. Jones wrote: > On Tue, Feb 15, 2022 at 05:24:14PM -0600, Eric Blake wrote: > > Oh. The QMP command (which is immediately visible through > > nbd-server-add/block-storage-add to qemu and qemu-storage-daemon) > > gains "multi-conn":"on", but you may be

Re: [PATCH v2] nbd/server: Allow MULTI_CONN for shared writable exports

2022-02-15 Thread Nir Soffer
On Tue, Feb 15, 2022 at 7:22 PM Eric Blake wrote: > According to the NBD spec, a server advertising > NBD_FLAG_CAN_MULTI_CONN promises that multiple client connections will > not see any cache inconsistencies: when properly separated by a single > flush, actions performed by one client will be vi

Re: [PATCH v2 2/2] iotests/block-status-cache: New test

2022-01-18 Thread Nir Soffer
we can only use the raw format > +iotests.main(supported_fmts=['raw'], > + supported_protocols=['file']) > diff --git a/tests/qemu-iotests/tests/block-status-cache.out > b/tests/qemu-iotests/tests/block-status-cache.out > new file mode 100644 > index 00..ae1213e6f8 > --- /dev/null > +++ b/tests/qemu-iotests/tests/block-status-cache.out > @@ -0,0 +1,5 @@ > +. > +-- > +Ran 1 tests > + > +OK > -- > 2.33.1 > The out file is not very useful, and even fragile - if the test framework will change the output format, the test will fail. Ideally we depend only on the relevant output of our tools, and using a different version of the test framework on replacing it (e.g pytest) will not require modifying the out files. Regardless I would like to see this fix merged and this issue already exists in other tests. Some tests in tests/ do have useful output that can make debugging failures easier. Reviewed-by: Nir Soffer

Re: [PATCH 1/2] block/io: Update BSC only if want_zero is true

2022-01-17 Thread Nir Soffer
data, and so we would update the block-status cache with wrong > information. > > Therefore, we should not update the cache with want_zero=false. > > Reported-by: Nir Soffer > Fixes: 0bc329fbb009f8601cec23bf2bc48ead0c5a5fa2 >("block: block-status cache for da

Re: [PATCH 2/2] iotests/block-status-cache: New test

2022-01-17 Thread Nir Soffer
nd after querying > ' + > + 'qemu:allocation-depth') > +print('Before:') > + print(map_pre) > +print('After:') > +print(map_post) > + > +self.fail("Map information differs") > + > + > +if __name__ == '__main__': > +# The block-status cache only works on the protocol layer, so to test it, > +# we can only use the raw format > +iotests.main(supported_fmts=['raw'], > + supported_protocols=['file']) > diff --git a/tests/qemu-iotests/tests/block-status-cache.out > b/tests/qemu-iotests/tests/block-status-cache.out > new file mode 100644 > index 00..ae1213e6f8 > --- /dev/null > +++ b/tests/qemu-iotests/tests/block-status-cache.out > @@ -0,0 +1,5 @@ > +. > +-- > +Ran 1 tests > + > +OK > -- > 2.33.1 > Reviewed-by: Nir Soffer

[PATCH] nbd/server.c: Remove unused field

2022-01-11 Thread Nir Soffer
NBDRequestData struct has unused QSIMPLEQ_ENTRY filed. It seems that this field exists since the first git commit and was never used. Signed-off-by: Nir Soffer --- nbd/server.c | 1 - 1 file changed, 1 deletion(-) diff --git a/nbd/server.c b/nbd/server.c index 3927f7789d..ce5b2a1d02 100644

Re: [PATCH 3/4] qemu-img: add --shallow option for qemu-img compare --stat

2021-09-29 Thread Nir Soffer
On Wed, Sep 29, 2021 at 7:28 PM Vladimir Sementsov-Ogievskiy wrote: > > 29.09.2021 19:00, Nir Soffer wrote: > > On Wed, Sep 29, 2021 at 4:37 PM Vladimir Sementsov-Ogievskiy > > wrote: > >> > >> Allow compare only top images of backing chains. That's usef

Re: [PATCH 3/4] qemu-img: add --shallow option for qemu-img compare --stat

2021-09-29 Thread Nir Soffer
On Wed, Sep 29, 2021 at 4:37 PM Vladimir Sementsov-Ogievskiy wrote: > > Allow compare only top images of backing chains. That's useful for > comparing two increments from the same chain of incremental backups. > > Signed-off-by: Vladimir Sementsov-Ogievskiy > --- > docs/tools/qemu-img.rst | 8 +

Re: [PATCH] qemu-nbd: Change default cache mode to writeback

2021-09-19 Thread Nir Soffer
On Mon, Aug 16, 2021 at 6:50 PM Eric Blake wrote: > > On Fri, Aug 13, 2021 at 11:55:19PM +0300, Nir Soffer wrote: > > Both qemu and qemu-img use writeback cache mode by default, which is > > already documented in qemu(1). qemu-nbd uses writethrough cache mode by > > defaul

[PATCH] qemu-nbd: Change default cache mode to writeback

2021-08-13 Thread Nir Soffer
the common use case. [1] https://bugzilla.redhat.com/1990656 Signed-off-by: Nir Soffer --- docs/tools/qemu-nbd.rst | 6 -- qemu-nbd.c | 6 -- 2 files changed, 8 insertions(+), 4 deletions(-) diff --git a/docs/tools/qemu-nbd.rst b/docs/tools/qemu-nbd.rst index ee862fa0bc..564

Re: [PATCH for-6.1?] iotest: Further enhance qemu-img-bitmaps

2021-07-21 Thread Nir Soffer
On 7/21/21 11:46 PM, Eric Blake wrote: Add a regression test to make sure we detect attempts to use 'qemu-img bitmap' to modify an in-use local file. Suggested-by: Nir Soffer Signed-off-by: Eric Blake --- Sadly, this missed my bitmaps pull request today. If there's any reason

Re: [PATCH v2 3/3] qemu-img: Add --skip-broken-bitmaps for 'convert --bitmaps'

2021-07-13 Thread Nir Soffer
On Tue, Jul 13, 2021 at 8:53 PM Eric Blake wrote: > > On Sat, Jul 10, 2021 at 09:37:35PM +0300, Nir Soffer wrote: > > > We don't want to delete inconsistent bitmaps by default: although a > > > corrupt bitmap is only a loss of optimization rather than a corruption &

Re: [PATCH v2 2/3] qemu-img: Fail fast on convert --bitmaps with inconsistent bitmap

2021-07-13 Thread Nir Soffer
On Tue, Jul 13, 2021 at 8:48 PM Eric Blake wrote: > > On Sat, Jul 10, 2021 at 09:06:24PM +0300, Nir Soffer wrote: > > On 7/9/21 6:39 PM, Eric Blake wrote: > > > Waiting until the end of the convert operation (a potentially > > > time-consuming task) to finally dete

Re: [PATCH v2 3/3] qemu-img: Add --skip-broken-bitmaps for 'convert --bitmaps'

2021-07-10 Thread Nir Soffer
On 7/9/21 6:39 PM, Eric Blake wrote: The point of 'qemu-img convert --bitmaps' is to be a convenience for actions that are already possible through a string of smaller 'qemu-img bitmap' sub-commands. One situation not accounted for already is that if a source image contains an inconsistent bitma

Re: [PATCH v2 2/3] qemu-img: Fail fast on convert --bitmaps with inconsistent bitmap

2021-07-10 Thread Nir Soffer
On 7/9/21 6:39 PM, Eric Blake wrote: Waiting until the end of the convert operation (a potentially time-consuming task) to finally detect that we can't copy a bitmap is bad, comparing to failing fast up front. Furthermore, this prevents us from leaving a file behind with a bitmap that is not mar

Re: [PATCH v2 1/3] iotests: Improve and rename test 291 to qemu-img-bitmap

2021-07-10 Thread Nir Soffer
itmap-remove to delete this bitmap from disk In this context a more useful error message would be: Try "qemu-img bitmap --remove" ... but this is not a new issue. +image: TEST_DIR/t.IMGFMT.copy +file format: IMGFMT +virtual size: 10 MiB (10485760 bytes) +cluster_size: 65536 +Format specific information: +bitmaps: +[0]: +flags: +name: b0 +granularity: 65536 +[1]: +flags: +[0]: auto +name: b4 +granularity: 65536 +corrupt: false *** done Reviewed-by: Nir Soffer

Re: [PATCH v3 3/2] qemu-img: Reword 'qemu-img map --output=json' docs

2021-07-07 Thread Nir Soffer
is optional; it is absent in >cases where ``human`` format would omit the entry or exit with an error. > -- > 2.31.1 Would be nice if this could be generated from the json schema instead of repeating the type and description of the fields, but this is a nice improvement. Reviewed-by: Nir Soffer

Re: [PATCH v2] docs: document file-posix locking protocol

2021-07-03 Thread Nir Soffer
On Sat, Jul 3, 2021 at 4:51 PM Vladimir Sementsov-Ogievskiy wrote: > > Let's document how we use file locks in file-posix driver, to allow > external programs to "communicate" in this way with Qemu. This makes the locking implementation public, so qemu can never change it without breaking externa

Re: [PATCH v2 2/1] qemu-img: Add "backing":true to unallocated map segments

2021-06-29 Thread Nir Soffer
On Tue, Jun 29, 2021 at 5:40 PM Kevin Wolf wrote: > > Am 29.06.2021 um 09:23 hat Vladimir Sementsov-Ogievskiy geschrieben: > > 28.06.2021 20:42, Eric Blake wrote: > > > On Wed, Jun 23, 2021 at 06:04:19PM +0200, Kevin Wolf wrote: > > > > > This is fine, but it means that this flag will present in a

Re: [PATCH v2 2/1] qemu-img: Add "backing":true to unallocated map segments

2021-06-23 Thread Nir Soffer
On Wed, Jun 23, 2021 at 7:04 PM Kevin Wolf wrote: > > Am 23.06.2021 um 15:58 hat Nir Soffer geschrieben: > > On Wed, Jun 23, 2021 at 11:58 AM Kevin Wolf wrote: > > > > > > Am 22.06.2021 um 18:56 hat Nir Soffer geschrieben: > > > > On Tue,

Re: [PATCH v2 2/1] qemu-img: Add "backing":true to unallocated map segments

2021-06-23 Thread Nir Soffer
On Wed, Jun 23, 2021 at 11:58 AM Kevin Wolf wrote: > > Am 22.06.2021 um 18:56 hat Nir Soffer geschrieben: > > On Tue, Jun 22, 2021 at 6:38 PM Kevin Wolf wrote: > > > > > > Am 11.06.2021 um 21:03 hat Eric Blake geschrieben: > > > > To save the user fro

Re: [PATCH v2 2/1] qemu-img: Add "backing":true to unallocated map segments

2021-06-22 Thread Nir Soffer
On Fri, Jun 11, 2021 at 10:03 PM Eric Blake wrote: > > To save the user from having to check 'qemu-img info --backing-chain' > or other followup command to determine which "depth":n goes beyond the > chain, add a boolean field "backing" that is set only for unallocated > portions of the disk. > >

Re: [PATCH v2 2/1] qemu-img: Add "backing":true to unallocated map segments

2021-06-22 Thread Nir Soffer
On Tue, Jun 22, 2021 at 6:38 PM Kevin Wolf wrote: > > Am 11.06.2021 um 21:03 hat Eric Blake geschrieben: > > To save the user from having to check 'qemu-img info --backing-chain' > > or other followup command to determine which "depth":n goes beyond the > > chain, add a boolean field "backing" tha

Re: [PATCH v2 2/1] qemu-img: Add "backing":true to unallocated map segments

2021-06-15 Thread Nir Soffer
On Tue, Jun 15, 2021 at 11:54 AM Vladimir Sementsov-Ogievskiy wrote: > > 11.06.2021 22:03, Eric Blake wrote: > > To save the user from having to check 'qemu-img info --backing-chain' > > or other followup command to determine which "depth":n goes beyond the > > chain, add a boolean field "backing"

Re: [PATCH 2/2] nbd: Add new qemu:joint-allocation metadata context

2021-06-14 Thread Nir Soffer
On Mon, Jun 14, 2021 at 4:56 PM Eric Blake wrote: > > On Sat, Jun 12, 2021 at 02:39:44AM +0300, Nir Soffer wrote: > > Since this change is not simple, and the chance that we also get the dirty > > bitmap included in the result seems to be very low, I decided to check the > &

Re: [PATCH 2/2] nbd: Add new qemu:joint-allocation metadata context

2021-06-11 Thread Nir Soffer
On Wed, Jun 9, 2021 at 9:01 PM Eric Blake wrote: > > When trying to reconstruct a qcow2 chain using information provided > over NBD, ovirt had been relying on an unsafe assumption that any > portion of the qcow2 file advertised as sparse would defer to the > backing image; this worked with what qe

Re: [PATCH] qemu-{img,nbd}: Don't report zeroed cluster as a hole

2021-06-11 Thread Nir Soffer
On Fri, Jun 11, 2021 at 9:34 PM Eric Blake wrote: > > On Fri, Jun 11, 2021 at 08:35:01PM +0300, Nir Soffer wrote: > > On Fri, Jun 11, 2021 at 4:28 PM Eric Blake wrote: > > > > > > On Fri, Jun 11, 2021 at 10:09:09AM +0200, Kevin Wolf wrote: > > > > >

Re: [PATCH v2] qemu-img: Make unallocated part of backing chain obvious in map

2021-06-11 Thread Nir Soffer
On Fri, Jun 11, 2021 at 5:59 PM Eric Blake wrote: > > On Fri, Jun 11, 2021 at 05:35:12PM +0300, Vladimir Sementsov-Ogievskiy wrote: > > > An obvious solution is to make 'qemu-img map --output=json' > > > distinguish between clusters that have a local allocation from those > > > that are found nowh

Re: [PATCH] qemu-{img,nbd}: Don't report zeroed cluster as a hole

2021-06-11 Thread Nir Soffer
On Fri, Jun 11, 2021 at 4:28 PM Eric Blake wrote: > > On Fri, Jun 11, 2021 at 10:09:09AM +0200, Kevin Wolf wrote: > > > Yes, that might work as well. But we didn't previously document > > > depth to be optional. Removing something from output risks breaking > > > more downstream tools that expec

Re: [PATCH] qemu-{img,nbd}: Don't report zeroed cluster as a hole

2021-06-11 Thread Nir Soffer
> ‫ב-11 ביוני 2021, בשעה 11:14, ‏‏Vladimir Sementsov-Ogievskiy > ‏ כתב/ה:‬ > > 11.06.2021 11:09, Kevin Wolf wrote: >> Am 10.06.2021 um 22:46 hat Eric Blake geschrieben: >>>> On Thu, Jun 10, 2021 at 11:09:05PM +0300, Nir Soffer wrote: >>>>>> But

  1   2   3   4   >