I thank you for all your work. I have since moved away from this block architecture and am no longer able to verify with an existing configuration.
On Fri, Nov 22, 2019 at 6:36 AM Timo Aaltonen <tjaal...@ubuntu.com> wrote: > Hello Patrick, or anyone else affected, > > Accepted libvirt into xenial-proposed. The package will build now and be > available at > https://launchpad.net/ubuntu/+source/libvirt/1.3.1-1ubuntu10.29 in a few > hours, and then in the -proposed repository. > > Please help us by testing this new package. See > https://wiki.ubuntu.com/Testing/EnableProposed for documentation on how > to enable and use -proposed. Your feedback will aid us getting this > update out to other Ubuntu users. > > If this package fixes the bug for you, please add a comment to this bug, > mentioning the version of the package you tested and change the tag from > verification-needed-xenial to verification-done-xenial. If it does not > fix the bug for you, please add a comment stating that, and change the > tag to verification-failed-xenial. In either case, without details of > your testing we will not be able to proceed. > > Further information regarding the verification process can be found at > https://wiki.ubuntu.com/QATeam/PerformingSRUVerification . Thank you in > advance for helping! > > N.B. The updated package will be released to -updates after the bug(s) > fixed by this package have been verified and the package has been in > -proposed for a minimum of 7 days. > > ** Changed in: libvirt (Ubuntu Xenial) > Status: In Progress => Fix Committed > > ** Tags added: verification-needed verification-needed-xenial > > -- > You received this bug notification because you are subscribed to the bug > report. > https://bugs.launchpad.net/bugs/1681839 > > Title: > libvirt: blockcommit fails - disk not ready for pivot yet > > Status in libvirt package in Ubuntu: > Fix Released > Status in libvirt source package in Xenial: > Fix Committed > Status in libvirt source package in Artful: > Fix Released > Status in libvirt source package in Bionic: > Fix Released > > Bug description: > [Impact] > > On xenial, if you manually invoke blockcommit through virsh in > libvirt, the command immediately fails with blockcommit supposedly > being 100%, and that the disk is not ready for pivot yet: > > root@xenial-apparmor:~# virsh blockcommit snapvm vda --active --verbose > --pivot --wait > Block commit: [100 %] > error: failed to pivot job for disk vda > error: block copy still active: disk 'vda' not ready for pivot yet > > However, if you look at the status of the active blockjob, we see that > the blockcommit is still active in the background: > > root@xenial-apparmor:~# virsh blockjob snapvm vda --info > Active Block Commit: [0 %] > > root@xenial-apparmor:~# virsh blockjob snapvm vda --info > Active Block Commit: [2 %] > > root@xenial-apparmor:~# virsh blockjob snapvm vda --info > Active Block Commit: [6 %] > > This happens until it reaches 100%, where it gets stuck. To un-stick > things, you must then manually --abort the blockjob. > > root@xenial-apparmor:~# virsh blockjob snapvm vda --info > Active Block Commit: [100 %] > > This happens in VMs which are experiencing load, and is caused by a > race condition in libvirt. Users are not able to commit their > snapshots to disk. > > [Test Case] > > Credit goes to Fabio Martins, who determined how to reproduce this > issue. > > On a Ubuntu 16.04 host with libvirt 1.3.1-1ubuntu10.27: > > 1) Create a VG and define a LVM pool: > > root@xenial-apparmor:~# cat lvmpool.xml > <pool type="logical"> > <name>LVMpool_vg</name> > <source> > <device path="/dev/sdb"/> > </source> > <target> > <path>/dev/LVMpool_vg</path> > </target> > </pool> > > # virsh pool-define lvmpool.xml > # virsh pool-start LVMpool_vg > # virsh pool-autostart LVMpool_vg > > 2) Create a config file to use as a cdrom device with the new VM (will > be created in next steps), just to inject a password with cloud-init: > > # cat > config <<EOF > > #cloud-config > > password: passw0rd > > chpasswd: { expire: False } > > ssh_pwauth: True > > EOF > > # apt install cloud-image-utils > > # cloud-localds config.img config > > # mv config.img /var/lib/libvirt/images/ > # chown libvirt-qemu:kvm /var/lib/libvirt/images/config.img > # chmod 664 /var/lib/libvirt/images/config.img > > 3) Create one VM using this pool: > > # virt-install --connect=qemu:///system --name snapvm --ram 2048 > --vcpus=1 --os-type=linux --disk pool=LVMpool_vg,size=15,bus=virtio > --disk /var/lib/libvirt/images/config.img,device=cdrom --network > network=kvm-br0 --graphics none --import --noautoconsole > > 4) Stop the VM > > # virsh destroy snapvm > > 5) Download a Ubuntu cloud image, convert to raw and restore it into > the LV used as a disk by our VM: > > # wget > https://cloud-images.ubuntu.com/bionic/current/bionic-server-cloudimg-amd64.img > # qemu-img convert ./bionic-server-cloudimg-amd64.img > ./bionic-server-cloudimg-amd64.raw > # dd if=./bionic-server-cloudimg-amd64.raw of=/dev/LVMpool_vg/snapvm > bs=8M conv=sparse > > 6) Start the VM and connect to it in another window > > # virsh start snapvm > > 7) Check that the VM is using the LV as the disk: > > root@xenial-apparmor:~# virsh domblklist snapvm > Target Source > ------------------------------------------------ > vda /dev/LVMpool_vg/snapvm > hda /var/lib/libvirt/images/config.img > > 8) Create a snapshot and check that the new domblklist points to the > snapshot file: > > # virsh snapshot-create-as --domain snapvm --diskspec > vda,file=/var/lib/libvirt/images/xenial-snapvm.qcow2,snapshot=external > --disk-only --atomic > > root@xenial-apparmor:~# virsh domblklist snapvm > Target Source > ------------------------------------------------ > vda /var/lib/libvirt/images/xenial-snapvm.qcow2 > hda /var/lib/libvirt/images/config.img > > 9) Connect to your VM and start an I/O intensive job. In this case I'm > starting a 'dd' writing zeroes to a file until it gets to 10GBs: > > ubuntu@ubuntu:~$ dd if=/dev/zero of=file.txt count=1024 bs=10240000 > > 10) Back to the host, monitor the snapshot file and let it grow until > at list a bit more than 1GB, as in the example below (where we can see > the file with 3.9G): > > root@xenial-apparmor:~# ls -lh /var/lib/libvirt/images/ > total 5.2G > -rw-rw-r-- 1 libvirt-qemu kvm 329M Sep 3 03:18 > bionic-server-cloudimg-amd64.img > -rw-r--r-- 1 root root 10G Sep 3 03:28 bionic-server-cloudimg-amd64.raw > -rw-rw-r-- 1 libvirt-qemu kvm 366K Sep 3 03:19 config.img > -rw------- 1 libvirt-qemu kvm 3.9G Sep 3 04:41 xenial-snapvm.qcow2 > > 11) Start a blockcommit job with --active --verbose --pivot --wait and > we'll hit the error when the job gets to 100%: > > root@xenial-apparmor:~# virsh blockcommit snapvm vda --active --verbose > --pivot --wait > Block commit: [100 %] > error: failed to pivot job for disk vda > error: block copy still active: disk 'vda' not ready for pivot yet > > 12) The blkjob will continue in the background, and status increments: > > root@xenial-apparmor:~# virsh blockjob snapvm vda --info > Active Block Commit: [0 %] > > root@xenial-apparmor:~# virsh blockjob snapvm vda --info > Active Block Commit: [2 %] > > root@xenial-apparmor:~# virsh blockjob snapvm vda --info > Active Block Commit: [6 %] > > 13) The blkjob will show it is stuck at 100% until you --abort the > blkjob: > > root@xenial-apparmor:~# virsh blockjob snapvm vda --info > Active Block Commit: [100 %] > > I have created a test package with the commits needed to solve the > problem, and it is available here: > > https://launchpad.net/~mruffell/+archive/ubuntu/sf242822-test > > What should happen: > > If you install the test libvirt-bin and libvirt0 packages from the > above ppa, and run through the test case, when blockcommit is invoked, > it will not fail immediately, and instead, will continue on until it > reaches 100%. Once 100% is reached, the blockjob will complete > successfully. > > [Regression Potential] > > While there are four commits which are required to fix this issue, all > of them are fairly minor and only modify the way the current status > percentage is counted, and how states are being changed, upon reaching > 100% blockcommit. All changes are localised to one file. > > Most of the commits are limited to blockcommit, and in event of > regression, only blockcommit and by extension, some blockjobs would be > impacted. > > The commits have been present in upstream for a long time, have been > well tested by the community, and are from a release of libvirt with > very small delta to the one in xenial (1.3.2 versus 1.3.1 in xenial), > I believe there is little risk of regression. > > [Other Info] > > The following commits were identified in the upstream bug: > https://bugzilla.redhat.com/show_bug.cgi?id=1197592 > > which are also listed in comment #6. > > commit 86c4df83b913dd73b79caeed2038291374384dc5 > Author: Michael Chapman <m...@very.puzzling.org> > Date: Wed Jan 27 13:24:54 2016 +1100 > Subject: virsh: improve waiting for block job readiness > > commit 8fa216bbb40df33e7fce5d727aa3dc334480878a > Author: Michael Chapman <m...@very.puzzling.org> > Date: Wed Jan 27 13:24:53 2016 +1100 > Subject: virsh: ensure SIGINT action is reset on all errors > > commit 15dee2ef24f2f19f6dcd30d997b81c8a14582361 > Author: Michael Chapman <m...@very.puzzling.org> > Date: Wed Jan 27 13:24:52 2016 +1100 > Subject: virsh: be consistent with style of loop exit > > commit 704dfd6b0fafe7eafca93a03793389239f8ab869 > Author: Michael Chapman <m...@very.puzzling.org> > Date: Wed Jan 27 13:24:51 2016 +1100 > Subject: virsh: avoid unnecessary progress updates > > These fix the problem, and were introduced in libvirt 1.3.2 upstream. > All commits are clean cherry picks, and the code is still present in > B, D, E and F. > > To manage notifications about this bug go to: > > https://bugs.launchpad.net/ubuntu/+source/libvirt/+bug/1681839/+subscriptions > -- You received this bug notification because you are a member of Ubuntu Bugs, which is subscribed to Ubuntu. https://bugs.launchpad.net/bugs/1681839 Title: libvirt: blockcommit fails - disk not ready for pivot yet To manage notifications about this bug go to: https://bugs.launchpad.net/ubuntu/+source/libvirt/+bug/1681839/+subscriptions -- ubuntu-bugs mailing list ubuntu-bugs@lists.ubuntu.com https://lists.ubuntu.com/mailman/listinfo/ubuntu-bugs
