On Thu, Feb 02, 2023 at 04:26:04PM +0000, Richard W.M. Jones wrote:
...
> > > $ time nbdkit -r curl
> > > https://cloud-images.ubuntu.com/jammy/current/jammy-server-cloudimg-amd64.img
> > > --filter=multi-conn multi-conn-mode=unsafe timeout=2000 --run ' nbdcopy
> > > --no-extents -p $uri jammy-server-cloudimg-amd64.img '
> >
> > Yay - I'm glad my multi-conn filter makes it easier to test things
> > like this!
> >
> > Should we tweak the docs in nbdkit-multi-conn-filter(1) to mention
> > that despite multi-conn-mode=unsafe being unsafe for a plugin that
> > does not have consistency, it is useful for timing tests on a plugin
> > where we suspect consistency is available in order to test timing to
> > see if it actually makes a difference?
>
> Yes, sounds good.
How about:
diff --git i/filters/multi-conn/nbdkit-multi-conn-filter.pod
w/filters/multi-conn/nbdkit-multi-conn-filter.pod
index 87b31692..7f70ade9 100644
--- i/filters/multi-conn/nbdkit-multi-conn-filter.pod
+++ w/filters/multi-conn/nbdkit-multi-conn-filter.pod
@@ -118,7 +118,11 @@ passed on to the plugin.
When B<unsafe> mode is chosen, this filter blindly advertises
multi-conn to the client even if the plugin lacks support. This is
dangerous, and risks data corruption if the client makes assumptions
-about flush consistency that were not actually met.
+about flush consistency that were not actually met. However, for a
+plugin that does not yet advertise multi-conn, but where it is
+suspected that the plugin behaves consistently, this is a great way to
+run timing and accuracy tests to see enabling multi-conn in the plugin
+will make a difference.
=item B<multi-conn-track-dirty=fast>
> > > Is there any work on adding multi-conn support to qemu's NBD client?
> >
> > Not that I'm aware of at the moment, but we have proof that it may
> > prove fruitful to have someone spend time on.
>
> Kubernetes team are complaining that this particular case (stream
> qcow2 file from a website and convert it to raw without a temporary
> file) is slow. This is not a case that I've encountered before
> because it's not relevant to virt-v2v, but it does appear to be
> important.
>
> - * - * -
>
> I didn't want to complicate the original message with irrelevant
> stuff, but there's something else I want to mention now. If we just
> use qemu's curl driver (thus eliminating NBD & nbdkit from the mix),
> it gets even slower:
>
> $ time ~/d/qemu/build/qemu-img convert -p -W -f qcow2 'json:{
> "file.readahead": 67108864, "file.driver": "http", "file.url":
> "http://oirase.annexia.org/tmp/jammy-server-cloudimg-amd64.qcow2";,
> "file.timeout":2000 }' -O raw jammy-server-cloudimg-amd64.img.raw
> (100.00/100%)
>
> real 3m53.923s
> user 0m13.751s
> sys 0m15.346s
>
> Now this is not something I'm personally concerned about (since I've
> long been arguing we should deprecate the qemu curl driver and use
> nbdkit), but it's also very surprising.
I'm also not surprised if qemu's curl driver is not efficient, because
it does not seem to be a case frequently used. Offloading it to other
paths, like nbdkit, is indeed a good goal.
--
Eric Blake, Principal Software Engineer
Red Hat, Inc. +1-919-301-3266
Virtualization: qemu.org | libvirt.org
_______________________________________________
Libguestfs mailing list
Libguestfs@redhat.com
https://listman.redhat.com/mailman/listinfo/libguestfs
