Seems a regression introduce in commit : (2.7.0)
commit 0965a41e998ab820b5d660c8abfc8c819c97bc1b
Author: Vladimir Sementsov-Ogievskiy
Date: Thu Jul 14 20:19:01 2016 +0300
mirror: double performance of the bulk stage if the disc is full
Mirror can do up to 16 in-flight requ
Another test that confirm the perf drop after the revert,
live-migration of disk of 51GB (raw written data, without hole)
between two host with nvme disk, 10Gb/s connectivity:
master qemu 1m14s (737 MB/s)
master qemu with reverted patch 2m30s (353 MB/s)
--
You received thi
I think the issue come from SEEK_HOLE call.
SEEK_HOLE is fine until we find a hole close to the offset,
It becomes a very expensive call when the HOLE is at the
end of file of a big file (or smaller fragmented file
because there is a lot of FS extent the driver should check.)
When we run a mirro
We don't have currently issue with qcow2 for now,
In code it goes on different path:
block/file-posix.c:3219:1:.bdrv_co_block_status = raw_co_block_status,
block/qcow2.c:5892:1: .bdrv_co_block_status = qcow2_co_block_status
But when I run strace on qcow2 live migration
It helps a lot, and it goes fast !
live block migration of 400GB raw disk:
master 1130s (362MB/s) qemu unstable/frozen
master+fix 445s (920MB/s) qemu stable
Thanks Max, It will be nice to have this one merged.
--
You received this bug notification because you are a member of qemu-
deve
Some more data for Qcow2,
Qcow2 format is not impacted by the issue because it
does not do SEEK_DATA/HOLE at all, because as you said it knows
from its own metadata.
But qcow2 instances with a RAW backing file does..
So qcow2 instances laverage also benefit of the patch,
if they have big RAW backi
Public bug reported:
We have odd behavior in operation where qemu freeze during long
seconds, We started an thread about that issue here:
https://lists.gnu.org/archive/html/qemu-devel/2020-11/msg05623.html
It happens at least during openstack nova snapshot (qemu blockdev-mirror)
or live block mi
t; > > resources to find the holes in the disk and don't let any for the VM ?
> >
> > > If you can provide the stack trace during the hang, we might be able to
> > > tell why we're even trying to find holes.
> >
> > > Please also provide your QEMU command line.
> >
> > Here the qemu command line :
> >
> > /usr/bin/qemu-system-x86_64 \
> > -name guest=instance-0034d494,debug-threads=on \
> > -S \
> > -object
> > secret,id=masterKey0,format=raw,file=/var/lib/libvirt/qemu/domain-9-instance-0034d494/master-key.aes
> > \
> > -machine pc-i440fx-bionic,accel=kvm,usb=off,dump-guest-core=off \
> > -cpu Broadwell-IBRS,md-clear=on,vmx=on \
> > -m 6 \
> > -overcommit mem-lock=off \
> > -smp 16,sockets=16,cores=1,threads=1 \
> > -uuid b959a460-84b0-4280-b851-96755027622b \
> > -smbios 'type=1,manufacturer=OpenStack Foundation,product=OpenStack
> > Nova,version=14.1.1,serial=5b429103-2856-154f-1caf-5ffb5694cdc3,uuid=b959a460-84b0-4280-b851-96755027622b,family=Virtual
> > Machine' \
> > -no-user-config \
> > -nodefaults \
> > -chardev socket,id=charmonitor,fd=28,server,nowait \
> > -mon chardev=charmonitor,id=monitor,mode=control \
> > -rtc base=utc,driftfix=slew \
> > -global kvm-pit.lost_tick_policy=delay \
> > -no-hpet \
> > -no-shutdown \
> > -boot strict=on \
> > -device piix3-usb-uhci,id=usb,bus=pci.0,addr=0x1.0x2 \
> > -drive
> > file=/home/instances/b959a460-84b0-4280-b851-96755027622b/disk,format=raw,if=none,id=drive-virtio-disk0,cache=none,discard=unmap,aio=native,throttling.iops-total=8
> > \
> > -device
> > virtio-blk-pci,scsi=off,bus=pci.0,addr=0x3,drive=drive-virtio-disk0,id=virtio-disk0,bootindex=1,write-cache=on
> > \
> > -add-fd set=1,fd=31 \
> > -chardev file,id=charserial0,path=/dev/fdset/1,append=on \
> > -device isa-serial,chardev=charserial0,id=serial0 \
> > -chardev pty,id=charserial1 \
> > -device isa-serial,chardev=charserial1,id=serial1 \
> > -device usb-tablet,id=input0,bus=usb.0,port=1 \
> > -vnc 10.224.27.81:0 \
> > -device cirrus-vga,id=video0,bus=pci.0,addr=0x2 \
> > -device virtio-balloon-pci,id=balloon0,bus=pci.0,addr=0x4 \
> > -s \
> > -sandbox
> > on,obsolete=deny,elevateprivileges=deny,spawn=deny,resourcecontrol=deny \
> > -msg timestamp=on
> >
> >
> > > At the moment, my assumption is that this is during a mirror block job
> > > which is migrating the disk to your destination server. Not looking for
> > > holes would mean that a sparse source file would become fully allocated
> > > on the destination, which is usually not wanted (also we would
> > > potentially transfer a lot more data over the networkj).
> >
> > The vm disk is already fully allocated, so in this case it doesn't change.
> >
> > > Can you give us a snippet from your strace that shows the individual
> > > lseek syscalls? Depending on which ranges are queried, maybe we could
> > > optimise things by caching the previous result.
> >
> > Here the strace during the block migration :
> >
> > strace -c -p 32571
> > strace: Process 32571 attached
> > ^Cstrace: Process 32571 detached
> > % timeseconds usecs/callcalls errors syscall
> > 94.15 13.754159 2503 5495 lseek
> > 3.47 0.507101 91 5549 sendmsg
> > 1.60 0.233516 84 2769 io_submit
> > 0.40 0.057817 11 5496 setsockopt
> > 0.18 0.025747 4 5730 184 recvmsg
> > 0.16 0.023560 6 4259 write
> > 0.02 0.002575 6 408 read
> > 0.02 0.002425 9 266 35 futex
> > 0.01 0.002136 12 184 ppoll
> > 0.00 0.000184 12 16 poll
> > 0.00 0.38 381 clone
> > 0.00 0.32 162 rt_sigprocmask
> > 0.00 0.13 131 ioctl
> > 100.00 14.60930330176 219 total
> >
> > > Also, a final remark, I know of some cases (on XFS) where lseeks were
> > > slow because the image file was heavily fragmented. Defragmenting the
> > > file resolved the problem, so this may be another thing to try.
> >
> > > On XFS, newer QEMU versions set an extent size hint on newly created
> > > image files (during qemu-img create), which can reduce fragmentation
> > > considerably.
> >
> > > Kevin
> >
> > > Server hosting the VM :
> > > - Bi-Xeon hosts With NVME storage and 10 Go Network card
> > > - Qemu 4.0 And Libvirt 5.4
> > > - Kernel 4.18.0.25
> > >
> > > Guest having the issue :
> > > - raw image with Debian 8
> > >
> > > Here the qemu img on the disk :
> > > > qemu-img info disk
> > > image: disk
> > > file format: raw
> > > virtual size: 400G (429496729600 bytes)
> > > disk size: 400G
> > >
> > >
> > > Quentin GROLLEAU
> > >
>
>
--
Alexandre Arents
