Re: [Libguestfs] [PATCH v3 03/14] nbd/server: Prepare for alternate-size headers

2023-05-31 Thread Vladimir Sementsov-Ogievskiy
On 30.05.23 19:29, Eric Blake wrote: On Mon, May 29, 2023 at 05:26:50PM +0300, Vladimir Sementsov-Ogievskiy wrote: On 15.05.23 22:53, Eric Blake wrote: Upstream NBD now documents[1] an extension that supports 64-bit effect lengths in requests. As part of that extension, the size of the reply h

Re: [Libguestfs] [PATCH v3 05/14] nbd: Add types for extended headers

2023-05-31 Thread Vladimir Sementsov-Ogievskiy
On 30.05.23 21:22, Eric Blake wrote: On Tue, May 30, 2023 at 04:23:46PM +0300, Vladimir Sementsov-Ogievskiy wrote: On 15.05.23 22:53, Eric Blake wrote: Add the constants and structs necessary for later patches to start implementing the NBD_OPT_EXTENDED_HEADERS extension in both the client and s

Re: [Libguestfs] [PATCH v3 06/14] nbd/server: Refactor handling of request payload

2023-05-31 Thread Vladimir Sementsov-Ogievskiy
On 15.05.23 22:53, Eric Blake wrote: Upcoming additions to support NBD 64-bit effect lengths allow for the possibility to distinguish between payload length (capped at 32M) and effect length (up to 63 bits). Without that extension, only the NBD_CMD_WRITE request has a payload; but with the exten

Re: [Libguestfs] [PATCH v3 07/14] nbd/server: Refactor to pass full request around

2023-05-31 Thread Vladimir Sementsov-Ogievskiy
On 15.05.23 22:53, Eric Blake wrote: Part of NBD's 64-bit headers extension involves passing the client's requested offset back as part of the reply header (one reason for this change: converting absolute offsets stored in NBD_REPLY_TYPE_OFFSET_DATA to relative offsets within the buffer is easier

Re: [Libguestfs] [libnbd PATCH] internal: Tweak use of attribute packed in union sbuf

2023-05-31 Thread Richard W.M. Jones
On Tue, May 30, 2023 at 09:10:13PM -0500, Eric Blake wrote: > While analyzing 'union sbuf' in preparation to add more members to the > union, I noticed several things related to __attribute__((packed)) > that can be improved. It helps to note that that the bulk of the > members of 'union sbuf' are

Re: [Libguestfs] [libnbd PATCH] internal: s/handle/cookie/ to match NBD spec

2023-05-31 Thread Richard W.M. Jones
On Tue, May 30, 2023 at 12:44:41PM -0500, Eric Blake wrote: > On Tue, May 30, 2023 at 01:18:55PM +0200, Laszlo Ersek wrote: > > On 5/29/23 18:24, Eric Blake wrote: > > > Externally, we have been exposing the 64-bit opaque marker for each > > > NBD packet as the "cookie", because it was less confusi

Re: [Libguestfs] [PATCH libguestfs 1/2] ocaml/implicit_close test: collect all currently unreachable blocks

2023-05-31 Thread Richard W.M. Jones
On Sat, May 27, 2023 at 03:32:36PM +0200, Jürgen Hötzel wrote: > Fixes failing implice_close test on OCaml 5. > --- > ocaml/t/guestfs_065_implicit_close.ml | 4 ++-- > 1 file changed, 2 insertions(+), 2 deletions(-) > > diff --git a/ocaml/t/guestfs_065_implicit_close.ml > b/ocaml/t/guestfs_065_i

Re: [Libguestfs] [PATCH libguestfs 2/2] Only leave/enter blocking_section when OCaml lock is not held

2023-05-31 Thread Richard W.M. Jones
On Sat, May 27, 2023 at 03:35:38PM +0200, Jürgen Hötzel wrote: > Fixes deadlocks on OCaml5 when trying to get the lock that is already > held: > > Fatal error during lock: Resource deadlock avoided > --- > ocaml/guestfs-c.c | 8 ++-- > 1 file changed, 6 insertions(+), 2 deletions(-) > > diff

Re: [Libguestfs] [PATCH libguestfs 1/2] ocaml/implicit_close test: collect all currently unreachable blocks

2023-05-31 Thread Laszlo Ersek
On 5/31/23 11:12, Richard W.M. Jones wrote: > On Sat, May 27, 2023 at 03:32:36PM +0200, Jürgen Hötzel wrote: >> Fixes failing implice_close test on OCaml 5. >> --- >> ocaml/t/guestfs_065_implicit_close.ml | 4 ++-- >> 1 file changed, 2 insertions(+), 2 deletions(-) >> >> diff --git a/ocaml/t/guest

Re: [Libguestfs] [libnbd PATCH] internal: Tweak use of attribute packed in union sbuf

2023-05-31 Thread Laszlo Ersek
On 5/31/23 04:10, Eric Blake wrote: > While analyzing 'union sbuf' in preparation to add more members to the > union, I noticed several things related to __attribute__((packed)) > that can be improved. It helps to note that that the bulk of the > members of 'union sbuf' are already marked packed s

Re: [Libguestfs] [PATCH libguestfs 1/2] ocaml/implicit_close test: collect all currently unreachable blocks

2023-05-31 Thread Richard W.M. Jones
This bug and thread seem relevant: https://github.com/ocaml/ocaml/issues/11812 https://discuss.ocaml.org/t/ocaml-5-gc-releasing-memory-back-to-the-os/11293 Rich. -- Richard Jones, Virtualization Group, Red Hat http://people.redhat.com/~rjones Read my programming and virtualization blog: http:

Re: [Libguestfs] [PATCH libguestfs 1/2] ocaml/implicit_close test: collect all currently unreachable blocks

2023-05-31 Thread Richard W.M. Jones
On Wed, May 31, 2023 at 12:32:49PM +0200, Laszlo Ersek wrote: > On 5/31/23 11:12, Richard W.M. Jones wrote: > > On Sat, May 27, 2023 at 03:32:36PM +0200, Jürgen Hötzel wrote: > >> Fixes failing implice_close test on OCaml 5. > >> --- > >> ocaml/t/guestfs_065_implicit_close.ml | 4 ++-- > >> 1 file

Re: [Libguestfs] [PATCH libguestfs 1/2] ocaml/implicit_close test: collect all currently unreachable blocks

2023-05-31 Thread Richard W.M. Jones
... And while I'm scrawling my throughts into this thread ... What we intend here are two slightly different operations: (A) Free every unreachable object. That's what we want in this specific place in the code. (B) Provide a soft test that the OCaml heap hasn't been screwed up because of bug

Re: [Libguestfs] [libnbd PATCH v3 03/22] protocol: Add definitions for extended headers

2023-05-31 Thread Laszlo Ersek
On 5/30/23 20:48, Eric Blake wrote: > On Tue, May 30, 2023 at 05:48:25PM +0200, Laszlo Ersek wrote: >> On 5/30/23 16:56, Wouter Verhelst wrote: >>> On Tue, May 30, 2023 at 01:50:59PM +0200, Laszlo Ersek wrote: On 5/25/23 15:00, Eric Blake wrote: >> > +struct nbd_structured_reply_block_stat

Re: [Libguestfs] [libnbd PATCH v3 04/22] states: Prepare to send 64-bit requests

2023-05-31 Thread Laszlo Ersek
On 5/30/23 20:18, Eric Blake wrote: > On Tue, May 30, 2023 at 04:06:32PM +0200, Laszlo Ersek wrote: >> On 5/25/23 15:00, Eric Blake wrote: >>> Support sending 64-bit requests if extended headers were negotiated. >>> This includes setting NBD_CMD_FLAG_PAYLOAD_LEN any time we send an >>> extended NBD

Re: [Libguestfs] [PATCH v3 08/14] nbd/server: Support 64-bit block status

2023-05-31 Thread Vladimir Sementsov-Ogievskiy
On 15.05.23 22:53, Eric Blake wrote: The NBD spec states that if the client negotiates extended headers, the server must avoid NBD_REPLY_TYPE_BLOCK_STATUS and instead use NBD_REPLY_TYPE_BLOCK_STATUS_EXT which supports 64-bit lengths, even if the reply does not need more than 32 bits. As of this

Re: [Libguestfs] [PATCH libguestfs 1/2] ocaml/implicit_close test: collect all currently unreachable blocks

2023-05-31 Thread Laszlo Ersek
On 5/31/23 13:13, Richard W.M. Jones wrote: > On Wed, May 31, 2023 at 12:32:49PM +0200, Laszlo Ersek wrote: >> On 5/31/23 11:12, Richard W.M. Jones wrote: >>> On Sat, May 27, 2023 at 03:32:36PM +0200, Jürgen Hötzel wrote: Fixes failing implice_close test on OCaml 5. --- ocaml/t/gues

Re: [Libguestfs] [PATCH libguestfs 1/2] ocaml/implicit_close test: collect all currently unreachable blocks

2023-05-31 Thread Richard W.M. Jones
On Wed, May 31, 2023 at 04:44:49PM +0200, Laszlo Ersek wrote: > On 5/31/23 13:13, Richard W.M. Jones wrote: > > On Wed, May 31, 2023 at 12:32:49PM +0200, Laszlo Ersek wrote: > >> On 5/31/23 11:12, Richard W.M. Jones wrote: > >>> On Sat, May 27, 2023 at 03:32:36PM +0200, Jürgen Hötzel wrote: >

Re: [Libguestfs] [PATCH v3 09/14] nbd/server: Initial support for extended headers

2023-05-31 Thread Vladimir Sementsov-Ogievskiy
On 15.05.23 22:53, Eric Blake wrote: Time to support clients that request extended headers. Now we can finally reach the code added across several previous patches. Even though the NBD spec has been altered to allow us to accept NBD_CMD_READ larger than the max payload size (provided our respon

Re: [Libguestfs] [PATCH libguestfs 1/2] ocaml/implicit_close test: collect all currently unreachable blocks

2023-05-31 Thread Laszlo Ersek
On 5/31/23 13:23, Richard W.M. Jones wrote: > > ... And while I'm scrawling my throughts into this thread ... > > What we intend here are two slightly different operations: > > (A) Free every unreachable object. That's what we want in this > specific place in the code. > > (B) Provide a soft t

Re: [Libguestfs] [PATCH v3 10/14] nbd/client: Initial support for extended headers

2023-05-31 Thread Vladimir Sementsov-Ogievskiy
On 15.05.23 22:53, Eric Blake wrote: Update the client code to be able to send an extended request, and parse an extended header from the server. Note that since we reject any structured reply with a too-large payload, we can always normalize a valid header back into the compact form, so that th

Re: [Libguestfs] [libnbd PATCH] internal: Tweak use of attribute packed in union sbuf

2023-05-31 Thread Eric Blake
On Wed, May 31, 2023 at 12:45:16PM +0200, Laszlo Ersek wrote: > On 5/31/23 04:10, Eric Blake wrote: > > While analyzing 'union sbuf' in preparation to add more members to the > > union, I noticed several things related to __attribute__((packed)) > > that can be improved. It helps to note that that

Re: [Libguestfs] [PATCH libguestfs 1/2] ocaml/implicit_close test: collect all currently unreachable blocks

2023-05-31 Thread Richard W.M. Jones
On Wed, May 31, 2023 at 05:33:13PM +0200, Laszlo Ersek wrote: > On 5/31/23 13:23, Richard W.M. Jones wrote: > > > > ... And while I'm scrawling my throughts into this thread ... > > > > What we intend here are two slightly different operations: > > > > (A) Free every unreachable object. That's

Re: [Libguestfs] [libnbd PATCH v3 04/22] states: Prepare to send 64-bit requests

2023-05-31 Thread Eric Blake
On Wed, May 31, 2023 at 01:59:12PM +0200, Laszlo Ersek wrote: > >> > >>> + > >>> + /* FIXME: future API addition to test if server negotiated extended > >>> mode. > >>> + * Until then, strict flags must ignore the PAYLOAD_LEN flag for > >>> pwrite, > >>> + * even though it is rejected for ot

Re: [Libguestfs] [libnbd PATCH v3 03/22] protocol: Add definitions for extended headers

2023-05-31 Thread Eric Blake
On Wed, May 31, 2023 at 01:29:30PM +0200, Laszlo Ersek wrote: > >> Putting aside alignment even, I don't understand why reducing "count" to > >> uint16_t would be reasonable. With the current 32-bit-only block > >> descriptor, we already need to write loops in libnbd clients, because we > >> can't

Re: [Libguestfs] [PATCH v3 11/14] nbd/client: Accept 64-bit block status chunks

2023-05-31 Thread Vladimir Sementsov-Ogievskiy
On 15.05.23 22:53, Eric Blake wrote: Because we use NBD_CMD_FLAG_REQ_ONE with NBD_CMD_BLOCK_STATUS, a client in narrow mode should not be able to provoke a server into sending a block status result larger than the client's 32-bit request. But in extended mode, a 64-bit status request must be able

Re: [Libguestfs] [PATCH v3 12/14] nbd/client: Request extended headers during negotiation

2023-05-31 Thread Vladimir Sementsov-Ogievskiy
On 15.05.23 22:53, Eric Blake wrote: All the pieces are in place for a client to finally request extended headers. Note that we must not request extended headers when qemu-nbd why must not? It should gracefully report ENOTSUP? Or not? is used to connect to the kernel module (as nbd.ko does n

Re: [Libguestfs] [PATCH v3 11/14] nbd/client: Accept 64-bit block status chunks

2023-05-31 Thread Eric Blake
On Wed, May 31, 2023 at 08:00:43PM +0300, Vladimir Sementsov-Ogievskiy wrote: > On 15.05.23 22:53, Eric Blake wrote: > > Because we use NBD_CMD_FLAG_REQ_ONE with NBD_CMD_BLOCK_STATUS, a > > client in narrow mode should not be able to provoke a server into > > sending a block status result larger th

Re: [Libguestfs] [PATCH v3 12/14] nbd/client: Request extended headers during negotiation

2023-05-31 Thread Eric Blake
On Wed, May 31, 2023 at 08:39:53PM +0300, Vladimir Sementsov-Ogievskiy wrote: > On 15.05.23 22:53, Eric Blake wrote: > > All the pieces are in place for a client to finally request extended > > headers. Note that we must not request extended headers when qemu-nbd > > why must not? It should grace

Re: [Libguestfs] [PATCH v3 11/14] nbd/client: Accept 64-bit block status chunks

2023-05-31 Thread Vladimir Sementsov-Ogievskiy
On 31.05.23 20:40, Eric Blake wrote: On Wed, May 31, 2023 at 08:00:43PM +0300, Vladimir Sementsov-Ogievskiy wrote: On 15.05.23 22:53, Eric Blake wrote: Because we use NBD_CMD_FLAG_REQ_ONE with NBD_CMD_BLOCK_STATUS, a client in narrow mode should not be able to provoke a server into sending a bl

Re: [Libguestfs] [PATCH v3 12/14] nbd/client: Request extended headers during negotiation

2023-05-31 Thread Vladimir Sementsov-Ogievskiy
On 31.05.23 20:54, Eric Blake wrote: On Wed, May 31, 2023 at 08:39:53PM +0300, Vladimir Sementsov-Ogievskiy wrote: On 15.05.23 22:53, Eric Blake wrote: All the pieces are in place for a client to finally request extended headers. Note that we must not request extended headers when qemu-nbd w

Re: [Libguestfs] [PATCH libguestfs 2/2] Only leave/enter blocking_section when OCaml lock is not held

2023-05-31 Thread Jürgen Hötzel
"Richard W.M. Jones" writes: > On Sat, May 27, 2023 at 03:35:38PM +0200, Jürgen Hötzel wrote: >> Fixes deadlocks on OCaml5 when trying to get the lock that is already >> held: >> >> Fatal error during lock: Resource deadlock avoided >> --- >> ocaml/guestfs-c.c | 8 ++-- >> 1 file changed,

Re: [Libguestfs] [PATCH v3 12/14] nbd/client: Request extended headers during negotiation

2023-05-31 Thread Eric Blake
On Wed, May 31, 2023 at 09:33:20PM +0300, Vladimir Sementsov-Ogievskiy wrote: > On 31.05.23 20:54, Eric Blake wrote: > > On Wed, May 31, 2023 at 08:39:53PM +0300, Vladimir Sementsov-Ogievskiy > > wrote: > > > On 15.05.23 22:53, Eric Blake wrote: > > > > All the pieces are in place for a client to

Re: [Libguestfs] [libnbd PATCH] internal: Tweak use of attribute packed in union sbuf

2023-05-31 Thread Laszlo Ersek
On 5/31/23 17:48, Eric Blake wrote: > On Wed, May 31, 2023 at 12:45:16PM +0200, Laszlo Ersek wrote: >> On 5/31/23 04:10, Eric Blake wrote: >>> While analyzing 'union sbuf' in preparation to add more members to the >>> union, I noticed several things related to __attribute__((packed)) >>> that can b

Re: [Libguestfs] [libnbd PATCH v3 03/22] protocol: Add definitions for extended headers

2023-05-31 Thread Laszlo Ersek
On 5/31/23 18:04, Eric Blake wrote: > On Wed, May 31, 2023 at 01:29:30PM +0200, Laszlo Ersek wrote: Putting aside alignment even, I don't understand why reducing "count" to uint16_t would be reasonable. With the current 32-bit-only block descriptor, we already need to write loops in

Re: [Libguestfs] [PATCH libguestfs 2/2] Only leave/enter blocking_section when OCaml lock is not held

2023-05-31 Thread Laszlo Ersek
On 5/31/23 20:29, Jürgen Hötzel wrote: > > "Richard W.M. Jones" writes: > >> On Sat, May 27, 2023 at 03:35:38PM +0200, Jürgen Hötzel wrote: >>> Fixes deadlocks on OCaml5 when trying to get the lock that is already >>> held: >>> >>> Fatal error during lock: Resource deadlock avoided >>> --- >>>