On Tue, Apr 21, 2015 at 11:34:07AM -0400, John Snow wrote:
> Kashyap Chamarthy sent me a question asking for some clarifications on
> testing my incremental backup series. I responded off-list but he has asked
> me to reproduce my answer for the archives :)
Thanks for the detailed response on both online/offline cases further
below.
Just as a side note, I wrote a short version of the same email to the
list to your v 2.5 thread. I think you sent this response before
noticing them. Sorry for the double emails. :-)
http://lists.nongnu.org/archive/html/qemu-devel/2015-04/msg02384.html
http://lists.nongnu.org/archive/html/qemu-devel/2015-04/msg02406.html
So for the record, your details below should give clues to answer those.
[. . .]
> >Test
> >----
> >
> >(1) I have the QEMU invocation w/ QMP of a CirrOS disk image, that way:
> >
> >
> > /home/kashyapc/tinker-space/qemu-upstream/x86_64-softmmu/qemu-system-x86_64
> > \
> > . . .
> > -drive
> > file=/export/full_backup.img,if=none,id=drive-ide0-0-0,format=qcow2 \
> > -qmp tcp:localhost:4444,server
> >
> >So, it's waiting on the server:
> >
> > $ ./2-min-qemu-invoc-qmp.sh
> > char device redirected to /dev/pts/31 (label charserial0)
> > QEMU waiting for connection on: disconnected:tcp:localhost:4444,server
> >
> >
> >(2) Then, when I execute the `drive-backup` QMP command:
> >-----------------------------------
> >#!/bin/bash
> >set -x
> ># Test incremental backup with `drive-backup`
> >
> >exec 3<>/dev/tcp/localhost/4444
> >echo -e "{ 'execute': 'qmp_capabilities' }" >&3
> >read response <&3
> >echo $response
> >echo -e "{ 'execute': 'drive-backup', 'arguments':
> > { 'device': 'drive-ide0-0-0', 'bitmap': 'bitmap0', 'sync':
> > 'dirty-bitmap', 'target':
> >
> > '/home/kashyapc/work/virt/qemu/incremental-backup-test-qemu/tests/incremental.0.img',
> > 'mode': 'existing', 'format': 'qcow2' } }" >&3
> >read response <&3
> >echo $response
> >-----------------------------------
> >
> >(3) Invoke it:
> >
> >$ ./invoke-drive-backup.sh
> >{"QMP": {"version": {"qemu": {"micro": 93, "minor": 2, "major": 2},
> >"package": ""}, "capabilities": []}}
> >{"return": {}}
> >
> >
> >Now, I don't see any feedback notification on the QMP server as a result
> >of the `drive-bacup` execution from step (3) unlike from a slightly old
> >test of regular `drive-backup` command[1].
> >
> >NOTE: I tested with and without pre-creating the 'incremental.0.img'
> >file, to no avail.
> >
> >Am I missing anything obvious?
> >
> >
> >[1] https://kashyapc.fedorapeople.org/virt/test-qmp-drive-backup.txt
> >
>
> (1) It doesn't appear as though you've created a bitmap in this example?
You're right! Shame on me for missing the important bit. I should have:
{ 'execute': 'block-dirty-bitmap-add', 'arguments': { 'node':
'drive-ide0-0-0', 'name': 'bitmap0' } }
> (2) Incremental backup is designed to work basically exclusively with
> 'mode': 'existing' because it relies on the user (or the management
> utility/api) to keep track of what the "last backup" was.
>
> (I had actually attempted to add that convenience feature into QEMU, but the
> sense was that the complexities of that were best left to libvirt, and I
> agreed.)
>
> Here's the workflow (if the VM has not begun guest execution)
> (A) Create a bitmap ("bitmap0" is my usual choice)
> (B) Create an "anchor" or "tail" backup:
> This is a full backup of the drive, performed as usual.
> (C) Create the image to be used, using e.g. tail.qcow2 as
> the backing file.
> (D) Ensure that <bitmap0> is clear by issuing a clear command. Look
> at bitmaps.md for an example.
> (E) You may optionally start the VM to allow the disk to change
> at this point.
> (F) You may now issue an incremental backup command:
> { 'execute': 'drive-backup',
> 'arguments': {
> 'device': 'drive-ide0-0-0',
> 'bitmap': 'bitmap0',
> 'sync': 'dirty-bitmap',
> 'target': .../tail.qcow2,
> 'mode': 'existing',
> 'format': 'qcow2'
> }
> }
>
> The workflow is different if the VM has already started,
> for data safety reasons. (We need extra precautions to ensure
> the coherency of the incremental backup.)
>
> (A) Create a bitmap. ("bitmap0")
> (B) Using a single transaction, you must accomplish the following:
> (i) Create a full backup of 'drive-ide0-0-0'
> (ii) Clear bitmap0.
> This ensures that the bitmap tracks only writes that have
> occurred after the full backup was made. This is why the
> transactions are important.
>
> It's OK to let QEMU create the tail.qcow2 file here,
> so we don't need mode:existing.
>
> { 'execute': 'transaction',
> 'arguments': {
> 'actions': [
> { 'type': 'block-dirty-bitmap-clear',
> 'data': {
> "node": 'drive-ide0-0-0',
> "name": 'bitmap0'
> }
> },
> { 'type': 'drive-backup',
> 'data': {
> 'device': 'drive-ide0-0-0',
> 'bitmap': 'bitmap0',
> 'sync': 'full',
> 'target': .../tail.qcow2,
> 'format': 'qcow2'
> }
> }
> ]
> }
> }
>
> (C) We need to create the incremental backup image outside of QEMU;
> # qemu-img create -f qcow2 -F qcow2 -b .../tail.qcow2
> (D) Now that we have a full backup and a synced bitmap, we can
> issue a standard incremental backup command at any time.
> { 'execute': 'drive-backup',
> 'arguments': {
> 'device': 'drive-ide0-0-0',
> 'bitmap': 'bitmap0',
> 'sync': 'dirty-bitmap',
> 'target': .../tail.qcow2,
> 'mode': 'existing',
> 'format': 'qcow2'
> }
> }
>
> (3) You are probably not seeing any return from this script because it looks
> as if you only read from the QMP stream once, which will just give you the
> reply from accepting the QMP command, but the event that tells you if it
> completed successfully or not occurs later, so you need one more read. If
> you use bash to send/receive QMP commands, you need to poll the fd to get
> event responses that occur later.
Thanks for these details! And, also didn't know of the "poll the fd"
bit you mention above below. As this shows, I just began poking at this,
so I'll obviously have to test more with different cases.
--
/kashyap
