On 2015-03-02 at 14:07, Max Reitz wrote:
On 2015-03-02 at 13:48, John Snow wrote:On 03/02/2015 12:49 PM, Max Reitz wrote:On 2015-02-27 at 19:47, John Snow wrote:Signed-off-by: John Snow <js...@redhat.com> --- docs/bitmaps.md | 303 ++++++++++++++++++++++++++++++++++++++++++++++++++++++++ 1 file changed, 303 insertions(+) create mode 100644 docs/bitmaps.md diff --git a/docs/bitmaps.md b/docs/bitmaps.md new file mode 100644 index 0000000..ebb6ae8 --- /dev/null +++ b/docs/bitmaps.md @@ -0,0 +1,303 @@ +# Dirty Bitmaps + +* Dirty bitmaps can be created at any time and attached to any node +(not just complete drives.) + +## Dirty Bitmap Names + +* A dirty bitmap's name is unique to the node, but bitmaps attached to different +nodes can share the same name. + +## Bitmap Modes ++* A Bitmap can be "enabled" (tracking writes, the default) or "disabled"+(read-only, I/O is ignored.) This state is currently only changed internally +for the purposes of migration, and otherwise remains enabled. + +* A Bitmap can be "frozen," which means that it is currently in-use by a backup +operation and cannot be deleted, enabled, disabled, renamed, written to, reset, +etc. + +## Basic QMP Usage + +### Supported Commands ### + +* block-dirty-bitmap-add +* block-dirty-bitmap-remove +* block-dirty-bitmap-clear + +### Creation + +* To create a new bitmap, enabled, on the drive with id=drive0: + +```json +{ "execute": "block-dirty-bitmap-add", + "arguments": { + "node": "drive0", + "name": "bitmap0" + } +} +``` + +* This bitmap will have a default granularity that matches the cluster size of +its associated drive, if available, clamped to between [4KiB, 64KiB]. +The current default for qcow2 is 64KiB. + +* To create a new bitmap that tracks changes in 32KiB segments: + +```json +{ "execute": "block-dirty-bitmap-add", + "arguments": { + "node": "drive0", + "name": "bitmap0", + "granularity": 32768 + } +} +``` + +### Deletion + +* Can be performed on a disabled bitmap, but not a frozen one. + +* Because bitmaps are only unique to the node to which they are attached, +you must specify the node/drive name here, too. + +```json +{ "execute": "block-dirty-bitmap-remove", + "arguments": { + "node": "drive0", + "name": "bitmap0" + } +} +``` + +### Resetting + +* Resetting a bitmap will clear all information it holds.+* An incremental backup created from an empty bitmap will copy no data,+as if nothing has changed. + +```json +{ "execute": "block-dirty-bitmap-clear", + "arguments": { + "node": "drive0", + "name": "bitmap0" + } +} +``` + +## Transactions (Not yet implemented) + +* Transactional commands are forthcoming in a future version, + and are not yet available for use. This section serves as + documentation of intent for their design and usage. + +### Justification+Bitmaps can be safely modified when the VM is paused or halted by using+the basic QMP commands. For instance, you might perform the following actions: + +1. Boot the VM in a paused state. +2. Create a full drive backup of drive0. +3. Create a new bitmap attached to drive0. +4. Resume execution of the VM. +5. Incremental backups are ready to be created. ++At this point, the bitmap and drive backup would be correctly in sync,+and incremental backups made from this point forward would be correctly aligned +to the full drive backup. + +This is not particularly useful if we decide we want to start incremental +backups after the VM has been running for a while, for which we will need to +perform actions such as the following: + +1. Boot the VM and begin execution. +2. Using a single transaction, perform the following operations: + * Create bitmap0. + * Create a full drive backup of drive0. +3. Incremental backups are now ready to be created. + +### Supported Bitmap Transactions + +* block-dirty-bitmap-add +* block-dirty-bitmap-clear ++The usages are identical to their respective QMP commands, but see below+for examples. + +### Example: New Incremental Backup + +As outlined in the justification, perhaps we want to create a new incremental +backup chain attached to a drive. + +```json +{ "execute": "transaction", + "arguments": { + "actions": [ + {"type": "block-dirty-bitmap-add", + "data": {"node": "drive0", "name": "bitmap0"} }, + {"type": "drive-backup",Note that this does not do a full drive-backup during the transaction but only starts the block job. Above you said you'd do a singletransaction in which you'd "Create a full drive backup of drive0" (whichis not possible right now, however). So the problem is that any write to the node while the block job isrunning will dirty the bitmap although they will be handled by the blockjob (which is not bad, your bitmap will simply be dirtier than it needs to be). I don't know exactly what Stefan proposed in regards to the callbacks, but I can imagine two solutions to this:It is my understanding that backup_run will register the backup_before_write_notify handler to back up any bits before they are written to, which means that the full backup makes a coherent backup _before_ any writes, and not a coherent backup _after_ all writes.Indeed, my mistake. I guess I finally found the difference between mirror and backup. :-)The dirty bitmap that results from this should describe new writes that occurred during the backup, but aren't reflected in the backup data.1. You add transaction support for completing a block job and clearing adirty bitmap. Then you'd simply start the drive-backup job, add the bitmap (both in any order), and then do a transaction of block-job-complete and block-dirty-bitmap-clear. But making block-job-complete safe for transactions can be difficult.2. You add a command to completely dirty a dirty bitmap (the opposite of block-dirty-bitmap-clear). This way, you'd simply create a dirty bitmap,mark everything dirty, and then you can start doing incremental backups from there (so you'd force a full backup in bitmap mode). Maybe that helps. Maybe it doesn't. I don't know.+ "data": {"device": "drive0", "target": "/path/to/full_backup.img", + "sync": "full", "format": "qcow2"} } + ] + } +} +``` + +### Example: New Incremental Backup Anchor Point + +Maybe we just want to create a new full backup with an existing bitmap and +want to reset the bitmap to track the new chain. + +```json +{ "execute": "transaction", + "arguments": { + "actions": [ + {"type": "block-dirty-bitmap-clear", + "data": {"node": "drive0", "name": "bitmap0"} }, + {"type": "drive-backup", + "data": {"device": "drive0", "target": "/path/to/new_full_backup.img", + "sync": "full", "format": "qcow2"} } + ] + } +} +```Same here, the dirty bitmap will be dirtier than it needs to be (notreally bad, but with the above suggestion we can do something about this).+ +## Incremental Backups + +The star of the show.I'm fine with a bit of fun in the documentation, but I'll see whether others are, too. :-) (I fear that maybe just being able to imagine someone not allowing fun is enough to reinforce the image of the no-fun-allowed German)+ +**Nota Bene!** Only incremental backups of entire drives are supported for now. +So despite the fact that you can attach a bitmap to any arbitrary node, they are +only currently useful when attached to the root node. This is because +drive-backup only supports drives/devices instead of arbitrary nodes.Well, not a real reason, since there's blockdev-backup, too. I'd just omit this here. If people see that you're using drive-backup, theyshould know that this can only be used for BlockBackends ("full drives").You can leave it, what I don't like is just that it sounds like it'll bereally difficult to implement it for single BDS nodes as well; but it's just a technical question of drive-backup vs. blockdev-backup. The core of everything, block/backup.c, should be agnostic of whether you useddrive-backup or blockdev-backup, so support is actually there, it's onlythe interface that's missing.I didn't mean to imply it wouldn't ever be useful, it's just factually not useful yet. I can try to rephrase this to make it clearer why attaching bitmaps to arbitrary nodes is not yet a useful thing to do.That would be nice, probably even better with a reference to blockdev-backup (that it exists but just does not have the necessary parameters yet).
Scratch that. My memory tricked me (am I getting old?), blockdev-backup does not work on node names.
Okay then, leave as-is, please (if it isn't too late already!). *cough* Max
