On Thu, Jan 23, 2020 at 07:53:57PM +0100, Max Reitz wrote: > On 23.01.20 19:35, Richard W.M. Jones wrote: > > - NBD multi-conn. In my tests this makes a really massive > > performance difference in certain situations. Again, virt-v2v has > > a lot of information that we cannot pass to qemu: we know, for > > example, exactly if the server supports the feature, how many > > threads are available, in some situations even have information > > about the network and backing disks that the data will travel over > > / be stored on. > > As far as I understand it, you use qemu-img convert with an NBD source > or target, too?
Virt-v2v has many modes, but yes generally there will be either an NBD source & target, or an NBD source to a local file target. > I suppose it’s always easier to let a specialized and freshly written > tool handle such information. But it sounds like if such information is > useful and makes that big of a difference, then it would be good to be > able to specify it to qemu’s NBD block driver, too. qemu-img convert has worked really well for us, and I'm actually _not_ confident that I could do better with a specialized tool. But there's definitely more info we could pass, such as the amount of parallelism we believe is available in the NBD server / processors / disks. > > - Machine-parsable progress bars. You can, sort of, parse the > > progress bar from qemu-img convert, but it's not as easy as it > > could be. In particular it would be nice if the format was treated > > as ABI, and if there was a way to have the tool write the progress > > bar info to a precreated file descriptor. > > It doesn’t seem impossible to add this feature to qemu-img, although I > wonder about the interface. I suppose we could make it an alternative > progress output mode (with some command-line flag), and then the > information would be emitted to stdout (just like the existing progress > report). You can of course redirect stdout to whatever fd you’d like, > so I don’t know whether qemu-img itself needs that specific capability. > > OTOH, if you need this feature, why not just use qemu itself? That is, > a mirror or a backup block job in an otherwise empty VM. I don't think we've really thought before about this approach. Maybe the launching of a VM (even an empty / stopped one) could be a problem. I guess this is what the new tool that was recently proposed upstream might help with? (Was it called qemu-block-storage? I can't find it right this minute) > > - External block lists. This is a rather obscure requirement, but > > it's necessary in the case where we can get the allocated block map > > from another source (eg. pyvmomi) and then want to use that with an > > NBD source that does not support extents (eg. nbdkit-ssh-plugin / > > libssh / sftp). [Having said that, it may be possible to implement > > this as an nbdkit filter, so maybe this is not a blocking feature.] > > That too seems like a feature that’s easily implementable in a > specialized tool, but hard to implement in qemu-img. > > I suppose we’d want a dirty bitmap copy mode which copies only the > regions that the bitmap reports as dirty – but at that point you’re > probably again better off not using qemu-img, but qemu itself. Then > we’d need some way to import bitmaps, and I actually don’t think we have > that yet. > > But again, if this is a generally useful feature, I think we want it in > qemu anyway. I think this is actually one we can more easily implement as an nbdkit filter. I'm going to try this and see. > > One thing which qemu-img convert can do which nbdcp could not: > > > > - Read or write from qcow2 files. > > > > So instead of splitting the ecosystem and writing a new tool that > > doesn't do as much as qemu-img convert, I wonder what qemu developers > > think about the above missing features? For example, are they in > > scope for qemu-img convert? > > What I think is that there may be features that we don’t want in > qemu-img, because they are more appropriate for the mirror or backup > block job. For example, I don’t think we want to let qemu-img convert > mess around with dirty bitmaps. > > But apart from that, the features you propose all seem useful to have in > qemu itself. Maybe some of them are too hard to implement (specifically > importing bitmaps from external sources), then it might be pragmatic to > write a new tool where such features can be easily implemented because > they don’t need to be integrated into an existing API. > > As for performance, well, if qemu’s NBD driver is slow, then naively I’d > think that’s a bug, isn’t it? And if improving performance requires > knobs, then that’s how it is. Thanks, Rich. -- Richard Jones, Virtualization Group, Red Hat http://people.redhat.com/~rjones Read my programming and virtualization blog: http://rwmj.wordpress.com virt-df lists disk usage of guests without needing to install any software inside the virtual machine. Supports Linux and Windows. http://people.redhat.com/~rjones/virt-df/
